Qualitative Evaluation of Facilitators’ Contributions to Online Professional Development

We relied on three core criteria for selecting facilitators for FOR-PD. First, the facilitator needed to have strong content knowledge in reading. Second, we sought online facilitators who had experience as literacy leaders and literacy experts. Third, we looked for facilitators who expressed desire to learn along with us about helping preK-12 teachers develop their reading knowledge and expertise. Given the novelty of this large-scale high profile state online project, prior experience with the Internet was not mandatory. However, we knew that some facilitators were reasonably comfortable with online education since they had been involved in it before as students or facilitators.

FOR-PD facilitators play a vital role in developing and maintaining an online professional development program that is effective, efficient, and supports the realization of the FOR-PD project objectives. The primary purpose of a FOR-PD facilitator is to interact with FOR-PD course participants. This translates to encouraging and replying to email messages and discussion postings, providing feedback on assignments, and being the “point person” for answering their questions. They must also be responsive to individual district requests and needs.

A facilitator in the online environment must possess a unique set of skills to perform effectively. Some of the basic criteria for a person to be successful as an online facilitator include the following:

• Facilitators must be able to create a supportive environment in which all students feel comfortable participating and especially where students know that their facilitator is accessible.

• Facilitators should give students timely quality feedback on student contributions to discussions, assignments, and quizzes.

• Facilitators should keep students advised of their progress respect to the course evaluation process on a regular basis

• Facilitators should feel comfortable communicating in writing. The face-to-face contact traditionally available in a classroom setting is not available in the online learning process. The ability to verbally communicate is replaced with a keyboard. Facilitators must be comfortable communicating in writing because that is the fundamental process of online learning. The facilitator is the primary person participants interact with who provides the human factor.

• Facilitators should be experienced and well trained in online learning. This includes sending and receiving email, using discussion boards, using chat tools, and using a Web browser.

To become a certified FOR-PD facilitator, interested educators must complete an online application and possess the following qualifications: (1) successful completion (80% mastery or above) of the FOR-PD course; (2) a minimum of three years teaching experience; (3) master’s degree in reading or other related areas; (4) advanced knowledge of research-based reading strategies; (5) ability to provide explicit instruction in the following elements of reading as they apply to appropriate grades: phonemic awareness, phonics, fluency, vocabulary, and comprehension; (6) ability to systematically use effective reading strategies that have been tested and have a record of success; and (7) identified by school or district as a reading/literacy leader. In addition to these requirements, successful completion of the FOR-PD Facilitator Training and Certification Course is also required. The FOR-PD Facilitator Training and Certification Course is a 25-hour online professional development course intended to ensure that our facilitators have the knowledge and skills they need to become successful online class facilitators for the Florida Online Reading Professional Development Course. The course consists of the five lessons that encompass an introduction to FOR-PD, details about the project and goals of FOR-PD, information on online learning, and support options to facilitators.

Following completion of the FOR-PD Facilitator Training and Certification Course, facilitators are expected to demonstrate mastery of the following skills: (1) describe the FOR-PD course and the goals of the course; (2) identify advantages of online learning; 3) identify potential disadvantages of online learning and describe at least one way each disadvantage can be addressed; (4) identify the role of the online class facilitator; (5) describe techniques for facilitating an online course; and (6) identify and use online tools such as chat, discussion boards, email, and grade books.

There is no charge to take the FOR-PD Facilitator Training and Certification Course. An electronic certificate (pdf) is emailed to participants upon successful completion, their district staff development office is notified, and they are then added to the pool of certified FOR-PD facilitators. Completion of the FOR-PD Facilitator Training and Certification course and certification as a FOR-PD facilitator does not guarantee employment as a facilitator. For the most part, facilitators are selected by school districts from the pool of qualified facilitators. Many school districts have a “favorite” facilitator or two who they assign to facilitate again and again. These are generally reading specialists, reading coaches, or literacy leaders in the district with particular knowledge of the unique qualities of the district, its reading programs, teachers, and student population.

Housed on the FOR-PD course server, the Facilitator Forum is a series of discussion boards offering 24/7 access for facilitators to interact with each other to share information and ideas about the FOR-PD course, to ask for help from others, and to share successes. Specific discussion areas include a place to meet fellow facilitators; to ask for and offer help, hints, and advice; to make suggestions; and to share success stories. The Facilitator Coffeehouse discussion board enables facilitators to interact with each other on matters unrelated to the FOR-PD course, but likely to be of general interest. Finally, there is a discussion area specifically for facilitators to discuss issues related to each of the 14 FOR-PD lessons.

The outside interim report of the first year of the FOR-PD project and course, drawn from various sources including narrative reports from facilitators, surveys of participants at the end of the course, and follow-up telephone interviews with administrators, reported the following:

• Over 87% of FOR-PD participants indicated they would make changes and/or additions to classroom reading instruction as a result of FOR-PD.

• Over 90% (93%) of participants indicated that the value of reading strategies introduced in FOR-PD was excellent or good.

• Approximately 97% of participants indicated FOR-PD was excellent or good in covering the state and national reading initiatives, with nearly three-fourths of participants (73%) indicating FOR-PD covered the reading initiatives to an excellent degree.

• Over 90% of participants indicated that FOR-PD has contributed to their knowledge of effective reading theory, research, and instructional practice to an excellent or good extent. The extent FOR-PD contributed to understanding student needs and instructional adaptations for struggling readers to an excellent or good extent was 89% with over one half indicating excellent (52%).

• Over 90% of participants rated the support from their facilitator as excellent (74%) or good (17%).

For the purpose of the qualitative evaluation of FOR-PD in phase II of the project, internal and external documents were reviewed. Hundreds of pages of qualitative data were collected and analyzed from the discussion boards of facilitators, with a focus on what contributions FOR-PD facilitators have made to the program, how they have experienced the program and, particularly, how they like the changes and revision of the FOR-PD course since summer 2003.

The site and major data source for the current qualitative evaluation was the facilitators’ discussion board postings from September 2003. There were 161 postings as of December 31, 2003, including topics such as: meeting peer facilitators and introducing their backgrounds; making comments; extending greetings to each other for the new (fall) semester 2003; offering help, hints, and advice; sharing success stories of FOR-PD participant learning; providing suggestions the FOR-PD facilitators’ electronic newsletters; providing general ideas and suggestions; posting messages related or unrelated to FOR-PD course; and discussing Lesson 1 to Lesson 14. The data used for the evaluation consisted of 120 messages posted by the facilitators out of the total of 161 on the discussion board since the beginning of phase 2. As the major source of data for the present evaluation, facilitators’ discussion board met some requirements of the RFP and a few focuses indicated in the UCF proposal for FOR-PD.

Robert Yin’s (1994) case study method was used for the design of this qualitative evaluation and one of its dominant modes, the combination of “pattern-matching” and “time-series analysis” was applied to analyze and explain facilitators’ perceptions and experiences of FOR-PD. Moreover, with the qualitative software Nvivo Revision [1.3], automatic coding of the data was used in addition to hand coding, and various codes were developed.

From the facilitators’ discussion board postings, two major categories emerged, messages conveying facilitators’ comments and messages conveying facilitators’ activities. Meanwhile, the frequency of postings on the discussion board varied largely from month to month at different data points: September, October, November, and December. As Table 1 shows, of the total 161 posts including coordinator or instructor’s messages, there are 95 more in the first half of the semester (September and October) than the latter half (November and December).

Operationally, “encouragement” included expressions like “look forward to new session,” “like the changes since summer,” “learn a lot” from FOR-PD, and so on, while “criticism” connoted “frustrating about participants who dropped out,” “unavailable assistance,” and so on. In terms of comments (see Table 2), the first half of the semester had a contrastingly larger amount of “encouragement” from facilitators than the latter half of the semester, with a ratio of 57 to 3. Likewise, 15 negative messages appeared in the first half, while none in the latter half. In general, there were many more “encouragement” messages than “criticism” messages during the whole section.

“Encouragement” messages were divided into six categories (Table 3). Specifically, many facilitators expressed cheerfulness, looked forward to new session, and liked the changes of layout and content since summer.

Following are quotes identified and extracted from the discussion board for “encouragement” messages from facilitators.

Table 3 also shows that, through the whole course, there were 15 messages in which facilitators provided “criticism,” all of which appeared in the first half of semester. Some conveyed facilitators’ confusion and frustration over technical problems, including those that might be caused by the change since summer, or facilitators’ uneasiness about participants’ low completion rate. As one facilitator noted, “It’s been frustrating because of participants who didn’t complete.” Moreover, a few facilitators expressed they could not reach administrative and technical support, such as the help desk.

One great use of the discussion forum was to ask for help, report to the FOR-PD program coordinator about problems and errors, and respond to or give suggestions to questions raised by other peer facilitators (see Table 4). Facilitators’ posts of activities, similar to their comments, were posted more frequently in the first half of the semester than in the latter half, with a ratio of 43 to 18.

An example of a comment from a facilitator asking for information or seeking help, including how they could solve problems for participants, follows:

Following is a series of posts concerning accessing quizzes which provides an example of reporting correspondence with FOR-PD UCF program coordinator and discussion with peer facilitators about problems and errors:

The findings from the facilitators’ discussion board showed several ongoing problems in the FOR-PD program. Following are a few recommendations to address them and a couple of suggestions for the final qualitative evaluation of FOR-PD program.

The help desk currently has a goal of addressing and resolving technical problems within a 24-hour period. Careful attention to continuing quick response should be monitored to ensure efficient and effective support in response to problems encountered by FOR-PD facilitators and participants.

Facilitators should not only be updated and familiarized before changes are made in the FOR-PD system to ensure they are comfortable with and understand how the changes will impact the course and can thereby be more effective in assisting participants, but they should also be reminded about the changes—for example, the changes of access to quizzes caused some confusion with facilitators.

While FOR-PD has been overall effective in using the online system, various technical problems have been frustrating to some participants and facilitators, which may or may not have been problems within FOR-PD itself. It is suggested that attention to improving the technical aspects of FOR-PD and researching and implementing ways to make the technology more user-friendly should be continued.

To understand the factors that have an impact on FOR-PD better, additional qualitative analyses should be conducted including telephone/online interviews and/or focus groups with five key audiences: school districts, participants, facilitators, Florida Department of Education staff, and FOR-PD content contributors or course designers/instructors.