DISTANCE EDUCATION TRAINING FOR DISTANCE EDUCATION TRAINERS: The Roadmap to Effective Distance Education Instructional Design Project

Institutions of higher learning are facing the challenge of offering support for technology-based faculty training and development efforts (Campus Computing Survey, 1999). More than a technological infrastructure is necessary to effectively encourage and train faculty members to teach at a distance. Other components, primarily focused on providing institutional support to assist a faculty member’s development, such as teaching incentives, instructional design support, and technology training, have been shown to be necessary in creating successful distance education training and development programs (Berge, 2001). Researchers have said instructors need new knowledge and skills to teach effectively by distance education (Beaudoin, 1990; Brigham, 1992; Dillon, Hengst, & Zoller, 1991; Shaeffer & Farr, 1993; Willis, 1993; Wolcott, 1993).

In addition, Spotts (1999) indicated that if instructors are expected to use instructional technologies—including distance education technologies—they need technical support and training. Instructional designers and technology specialists need to be knowledgeable about not only the latest technology, but also the educational methods to use that technology, according to Telg (1995) and Irani and Telg (2001). A study of14 land-grant universities (Irani & Telg, 2001) found that nearly two thirds (61.5%) of distance education faculty training was conducted by staff instructional designers—with no faculty appointment. Also, 64.3% of instructional designers who actively work with faculty to develop distance education courses had had no prior training or knowledge of instructional design methods used in distance education before working at their universities. Twelve of the 14 respondents said they had learned distance education instructional design methods while “on the job.”

These findings mirror a previous study of video production specialists who support their universities’ distance education effort. Telg (1995) found that many video production specialists had learned distance education instructional methods while on the job. Telg (1996a) recommended that a training curriculum be developed to teach television-production-specialist-turned-instructional-designers the information and skills—particularly knowledge of instructional design—that they needed to perform their jobs, so they can subsequently support faculty members’ efforts. Particular areas of instructional design in which video producers needed more knowledge included audience identification and needs, adult education theory, adapting content to the technology, distance education theory, interaction methods in distance education, and evaluation techniques in distance education (Telg, 1996b). Instructional designers must be adequately prepared in order to assist faculty, so that faculty can effectively teach undergraduate and graduate distance courses. However, these two questions still remain: Who provides the technology skills and instructional design training and support for faculty? And are these staff members adequately prepared to train and support the faculty?

In response to this need, six universities— the University of Florida, Texas A&M University, Texas Tech University, the University of Idaho, the University of Missouri-St. Louis, and Iowa State University—collaborated on a project titled Roadmap to Effective Distance Education Instructional Design. This project was funded by a U.S. Department of Agriculture Higher Education Grant to develop effective materials and innovative approaches to better prepare instructional designers at land-grant universities and other universities with agricultural academic programs to support their universities’ distance education teaching programs. The project development team partnered with the Association for Communication Excellence in Agriculture, Natural Resources, and Life and Human Sciences (ACE) and the American Distance Education Consortium (ADEC). These two organizations encompass most of the instructional designers working in distance education in the land-grant system. This “train-the-trainer” approach provided distance education instructional designers—who may be learning instructional design theory and practice on the job—with skills and knowledge to more effectively help faculty members develop distance education courses.

This case study describes the development process undertaken from August 2002 through July 2004 to carry out the Roadmap to Effective Distance Education Instructional Design project, in an attempt to explore its evolution and provide a model for development of similar programs at other institutions engaged in distance education. As such, the methodology utilized was an exploratory case study research design that combined observation and data collection in various forms (Berg, 2001). Case study methods involve a systematic approach to gathering information about some phenomenon so as to assist the researcher’s understanding of how it operates or functions. As such, case study methodology is not a form of data collection in the accepted sense of the term, but is a methodological approach that can involve several different types of data (Hamel, Dufour, & Fortin, 1993).

Roadmap’s development process included three phases—needs assessment, implementation, and evaluation—and collaborative efforts among institutions. The needs assessment determined the need for the project and the content that would be covered. The implementation phase consisted of content development, marketing, and content delivery. The evaluation phase, which overlapped the implementation phase, consisted of program evaluation and participant evaluation.

Before the project was started, a needs assessment survey was sent to ACE and ADEC member listservs, to help identify key characteristics of this virtual training project. Much of the actual program design was based on the needs assessment results. Twenty-four people responded to the needs assessment. Respondents were generally interested in participating, saw the project as useful, said they would have the time to complete the training program, wanted to be certified as effective instructional designers upon completion of the program, and preferred asynchronous delivery methods. Respondents’ position titles included training specialists, administrators, information technologists, instructors, academic deans, faculty members, extension agents, and Web developers.

Respondents were also asked what training they felt they were most in need of in the areas of instructional design and development. Their responses included designing asynchronous learning, designing content for adult learners, distance education best practices, and creating manageable segments/modules for learners. When asked about previous training in distance education instructional design, training, and development, most had had some technology and software training, but said that instructional design principles were self-taught.

Following the needs assessment, implementation of the project began. The implementation phase consisted of content development, marketing, and content delivery. Evaluation also was done through several steps in the implementation phase so that project designers received input on the materials that were being developed. Evaluation results will be provided in the next section.

The project team developed six content modules, called destinations to go with the Roadmap theme (see Table 1). Content was designed and delivered in WebCT. In addition to the development of the destinations, the University of Florida served as the coordinating institution, Iowa State hosted the WebCT and streaming video servers, and Texas A&M conferred a certification of completion at the end of the project. The project team met every month via desktop videoconferencing using a Polycom bridge. More of this process will be discussed in the “Collaborative Efforts Among Institutions” section. All collaborators helped each other identify references and resources for each destination’s content to be covered. Collaborating institutions pilot-tested most destinations.

As the content was being developed, a marketing campaign, using the ACE and ADEC listservs, was undertaken concurrently to promote the Roadmap program. Promotional pieces, including a Web site and brochure, were developed, and a session was presented at the 2002 ACE Conference to promote Roadmap. In addition, each collaborating institution was allotted 10 “slots” to fill (60 total). All remaining slots were to be filled on a first-come, first-reserved basis. A total of 96 people, representing 26 institutions, participated in Roadmap (see Table 2). Since the USDA grant underwrote all development and delivery costs, participants took the course at no charge; those completing all requirements received a certificate of completion from Texas A&M’s Office of Distance Education. The only cost associated with the project was if participants wished to receive continuing education credits (CEUs); a nominal fee of $17 was assessed for the CEUs.

Each individual destination featured different synchronous or asynchronous delivery methods to provide participants with various examples of how to deliver educational materials at a distance. Destinations featured streaming video files and narrated PowerPoint presentations, in addition to the asynchronous content. Participants also communicated with each other and with project developers in chat rooms, threaded discussion boards, and emails in WebCT. In addition, Web-based training materials—PDF files, video clips, and related materials—were provided to the participants to use in the training of their own faculty members.

At the conclusion of each destination, participants were to complete a microproject—a short assignment designed to show that the participants understood and could apply the content that was presented. Participants had 3 weeks to complete the content and the microproject. During the fourth week, the destination coordinators evaluated the microprojects and returned their comments. The team member responsible for a particular destination was responsible for evaluating the microproject and providing feedback to each participant submitting a microproject. After the microproject was critiqued, the next destination would start.

There were two components to the evaluation phase: evaluation of the participants (including certification) and evaluation of the project. Participant and program evaluation were conducted several times throughout the project. Participant evaluation consisted of a pre-/post-self-assessment of participants’ knowledge of core competencies and their mastery of these competencies and concepts, as evidenced by their completion of microprojects. Program evaluation was collected in several stages—formative evaluations during the orientation and after the third destination, focus groups, and a summative evaluation at the project’s conclusion.

During the week-long orientation session, participants were asked to complete a series of questions that indicated their perceptions of how well they knew about or had mastered the core distance education instructional design competencies, identified in previous research (Dooley & Lindner, 2002). In this study, competencies refer to the group of related knowledge, skills, and abilities that affect participants’ in their jobs related to distance education and instructional design. The competencies used in this study had been examined before by Dooley and Lindner (2002) and were developed around the competencies for the distance education professional (Thach & Murphy, 1995). The competencies consisted of the following areas: adult learning, teaching at a distance, instructional design, course development, delivery strategies for teaching at a distance, distance and instructional technology resources, advanced interaction methods, accessibility, planning and conducting evaluation, evaluation analysis and reporting, administrative issues, and training and support. Previous studies on the use of self-assessment tools to document growth or learning had been conducted with graduate students through a semester-long course (Dooley & Lindner, 2002) and international professionals during a week-long training program (Dooley, Lindner, Dooley, & Algeera, 2004), so this framework served as the guidepost for the design for this study. Project developers wanted to determine if knowledge and skills were acquired by these professionals as a result of the Roadmap training program.

The self-assessment instrument was intended to serve as a reflection tool for the participants to measure their growth (learning) in the core competencies areas. The instrument uses a stair-step approach, rather than a continuum or Likert scale, to visually represent progression from novice (0) to expert (7). The numbers were intended to measure perceived growth rather than any statistical significance. Averages were calculated to show trends in the data only. Participants were asked to verify their answers with a brief narrative. Responses were coded to ensure confidentiality. The constant comparative method was used for data analysis on the verification narrative (Lincoln & Guba, 1985). An example of the self-assessment instrument for Destination 1 is provided in Figure 1.

At the end of the course, a summative assessment was conducted based on the pre-test/postcomparison of the competencies. Thirty-five participants completed both the pre-and post-assessment. In every case, respondents reported significant growth in each of the specific competencies (see Table 3).

The destinations’ microprojects reflected each core competency. For example, participants were asked to develop and report their learning philosophy for Destination 1, write objective statements for the lesson they would teach using distance technologies for Destination 2, determine what technology to use to teach a distance course for Destination 3, integrate interactive learning methods for Destination 4, develop an assessment tool for Destination 5, and identify effective components of distance courses or programs for Destination 6. These effective components from Destination 6’s microproject were used to develop an “exemplar database” of distance education programs. Fifty-five of the original 96 completed all six microprojects and received the certificate from Texas A&M. Twenty-three participants completed at least one of the first three destinations, 18 did not complete any destination but did log-on to view the content, and 10 never logged into the course.

Program evaluation data was collected at multiple times. An initial assessment (formative evaluation) was conducted of course participants at the end of the orientation module to gauge their time expectations for participating in the course, their rationale for participating, what they thought should be the priorities for instruction, their expectations for participating in the program, and their expectations for course outcomes. This assessment took the form of an online questionnaire delivered via Flashlight™ Online, a Web-based system for creating surveys, gathering responses, and analyzing data.

Researchers stress the importance of understanding the backgrounds and needs of students in designing effective distance education (Kemp, Morrison, & Ross, 1994; Threlkeld & Brzoska, 1994). According to Threlkeld and Brzoska (1994), this information is helpful in curriculum decisions, program design, and anticipation of student learning in distance education. According to Kirkpatrick (1996), an assessment of student reactions to and student learning in an educational program should precede assessment of behavior changes and learning outcomes.

Eighty participants (78%) responded to the formative evaluation given at the end of the orientation module in September 2003. Ninety-five percent of respondents (n = 76) said they were participating in the course to improve their skills and develop professionally. Other reasons cited by the respondents for participating in the Roadmap course were “The subject matter looked interesting” (n = 33), and “The way it was offered was convenient” (n = 28). Only a small portion of the respondents (n = 3) reported participating in the course in order to fulfill a continuing education requirement. According to respondents, the number one priority for the program should be for participants to “apply what they are learning to ‘real world’ situations.”

Respondents expected to be able to apply the principles of instructional design and adult learning to distance education course development. Respondents also expected to integrate a variety of delivery strategies and technologies in distance education course design. Additionally, respondents indicated their expectation to be able to incorporate opportunities for interaction into distance education courses (see Table 4).

Focus groups were conducted with course participants at three of the collaborating institutions at the end of the third destination. Focus groups requested more activities and less reading. They also asked for a one-page fact sheet or summary for each destination to be provided at the end of the module. Finally, participants indicated that they would like more interaction with other participants and would like to be able to work through all the destinations at their own pace, rather than being constrained by certain time periods for each destination. Several of these suggestions were incorporated into later destinations.

After the first three destinations in the Roadmap sequence were delivered, participants were asked, via e-mail, to describe their participation and expectations in the first three destinations. Twenty-six participants responded. Many respondents noted demanding schedules and time conflicts, saying they were unable to participate as much as they would like to. One participant described participation as being “quite active, but in short flurries.” Other participants indicated they completed the final project for each destination, but expressed regret in lacking time to interact with others. Several respondents said they expected to learn from being a student in a distance education course; they were used to designing and delivering instruction, not receiving it. Other respondents expressed their desire to develop their understanding of distance education and learn techniques in instructional design. Many respondents indicated their expectation to gain more in-depth knowledge in distance education, instructional design, and instructional technology. Most of the respondents indicated the Roadmap course was meeting their expectations. One respondent said, “Being on the receiving end of this course has given me a new perspective on what a student goes through— both the good and bad.” Another respondent said the course “exceeded my expectations because I didn’t expect as much depth.”

A summative evaluation was conducted during Destination 5 to determine the program’s overall effectiveness. This descriptive survey instrument, created in Flashlight™ Online, contained 45 items that collected information regarding participants’ experiences with and participation in the course, perceptions of course priorities, confidence in their ability to apply course concepts and skills, plans for using what they learned in the course, satisfaction with the content, delivery, administration, communication, and level of interaction in the course, and suggestions for improving the course. This instrument was reviewed prior to use by a panel of experts. To assess reliability of the final instrument, Cronbach’s alpha statistic was calculated, resulting in a standardized item alpha of .84 for the overall scale. There were 51 responses to the summative assessment.

Respondents were asked whether they were using any of the information from the Roadmap course in their job. Many respondents reported using copies of the materials as handouts to give to faculty and in workshops. Additionally, several respondents said they used the information in writing proposals for grants or new courses. They also reported using the information in designing their own courses or courses for other faculty members, as well as in teaching. One respondent said, “I have tried to take a look at my adult learners and be more sensitive to their needs, skill levels, and technology fears.” Respondents noted that they had used what they learned in Roadmap in their jobs (M = 4.01, SD = 1.00, on a 5-point scale where “1 = strongly disagree” and “5 = strongly agree”), the content was useful (M = 4.19, SD = .44), and the content related to their professional work (M = 4.34, SD = .77). Respondents also were positive about their Roadmap experience (see Table 5).

Participants also were asked a series of open-ended questions, designed for them to describe their Roadmap experience, in terms of satisfaction and usefulness. The authors analyzed these looking for common themes and relevant quotes. Common themes used to describe the Roadmap experience included informative, challenging, practical, useful, beneficial, demanding, and rewarding. One participant wrote:

With respect to satisfaction with the content and its usefulness, the majority of respondents said they were satisfied with the content and the experience and felt it was highly relevant to their needs. As one participant noted:

Finally, participants commented on the unique aspect of using distance education delivery media to provide content on how to develop distance education. As one participant wrote, “Life imitated art. Having a course on how to develop distance education materials delivered in the same format was a stroke of genius.”

Throughout the project, collaborating institutions communicated frequently and shared resources often. The primary collaborating method was e-mail. However, each month the project team met for one hour via desktop computer videoconferencing—using a Polycom bridge—to discuss development issues: recaps of past destinations, updates on upcoming destinations, budget concerns, conference papers and presentations, and evaluation issues. Also, several team members met at research and professional conferences throughout the year. During these conferences, team members tried to carve out at least an hour for the team to meet face-to-face to discuss the development, implementation, and evaluation of the project.

Participants in the project had “self-taught” instructional design principles at their institutions; Roadmap to Effective Distance Education Instructional Design—a train-the-trainer model—sought to teach instructional designers and technologists to be more adequately prepared to assist faculty who, in turn, can teach undergraduate and graduate distance courses more effectively. As evidenced by the evaluations and microprojects, this goal was achieved.

Participants’ self-assessments of their growth in learning, verified through their narratives, indicated the participants had learned and been able to apply the content in Roadmap. As noted earlier, competency indicators and self-assessments were used to serve as a reflection tool for participants. Roadmap collaborators recommend that others developing similar projects with professionals consider using competencies and self-assessment instruments, instead of traditional test questions. The competency-based assignments, evaluated by the completed microprojects, and self-assessment approach allow participants— particularly professionals—with the opportunity to apply and reflect on what they have learned.

As part of the train-the-trainer model, project collaborators tried to model good instructional design throughout the life of Roadmap. Collaborators followed an instructional design framework of needs assessment, audience identification, content identification and development, course/program design, program delivery, assignment development, and evaluation (formative and summative assessments). Collaborators wanted to “practice what we preached” to this group of professionals. One way this was done was to integrate revisions into the project after the second formative evaluation and the focus groups. Partic ipants said they wanted more interaction with fellow participants. Collaborators designing the remaining three destinations provided more opportunities—through chat rooms, email, and discussion boards—for participants to interact. This helped build a community of learners. Collaborators also recommend that others developing similar projects model good instructional design for their intended audiences.

Based on this collaborative effort of six universities (University of Florida, Texas A&M University, Texas Tech University, Iowa State University, the University of Idaho, and the University of Missouri-St. Louis) with well-recognized and respected distance education programs, Roadmap to Effective Distance Education Instructional Design will raise the level of the type of work done by distance education instructional designers. The certification process played and will continue to play a major role in helping distance education instructional designers at land-grant institutions raise their own stature for the positions they hold at their respective universities.

Finally, the project development team offers these insights and recommendations for others interested in conducting similar collaborative multi-institutional projects:

Start early. The initial plan was to start delivering content in April 2003. The project team then opted to delay the start date until September 2003. This extra time was needed to adequately design the content of the first two destinations.

Hire a coordinator. The University of Florida hired a project coordinator whose responsibility was to maintain communication, oversee the project development, and be the “point person” for participants’ questions.

Look for ways to improve the project. The team began improving the project as Roadmap was being delivered. Afterward, the team redesigned the “look” of the materials, divided the content into smaller modules, and is in the process of testing the new 11-module Roadmap.

Communicate frequently. The e-mail and videoconference communications among collaborators and face-to-face meetings at research and professional conferences were essential to the success of this project.

This article is based on a collaborative effort of all of the authors listed. The authors and contributors are involved heavily in the development and research of distance education at their respective institutions. Among them, the authors and contributors have received awards and honors for their distance education-related research and for distance education courses they have designed. This project was funded by a U.S. Department of Agriculture Challenge Grant.

Nick Place, Associate Professor, University of Florida

Colleen Swain, Associate Professor, University of Florida

Lori Moore, Assistant Professor, University of Idaho

Jimmy Lindner, Associate Professor, Texas A&M University

Allan Schmidt, Instructional Technologist, Iowa State University

Richard Carter, Professor, Iowa State University

Cindy Akers, Assistant Professor, Texas Tech University

Chad Davis, Assistant Professor, Texas Tech University

Clark Hickman, Assistant Professor, University of Missouri-St. Louis