This study was designed to measure instructional design competencies as a result of participation in a 9-month Web-based training program called Roadmap to Effective Distance Education Instructional Design. The researchers used a self-assessment pre- and posttest to determine participant initial and final competence in 12 areas: adult learning, understanding teaching at a distance, instructional design, course development, delivery strategies for teaching at a distance, instructional technology resources, advanced interaction methods, accessibility, planning and conducting evaluation, evaluation analysis and reporting, administrative issues, and training and support. Open-ended verification narratives were analyzed using the constant comparative method. This competency model worked well to document learning as a result of participation in the program.
INTRODUCTION
Institutions of higher learning are facing the challenge of offering support for technology-based faculty training and development efforts (Campus Computing Survey, 1999). Institutional support to assist a faculty member’s development, such as teaching incentives, instructional design support, and technology training, has been shown to be necessary in creating successful distance education training and development programs (Berge, 2001) so that instructors will have the new knowledge and skills that researchers indicate are necessary to teach effectively by distance education (Beaudoin, 1990; Brigham, 1992; Dillon, Hengst, & Zoller, 1991; Shaeffer & Farr, 1993; Willis, 1993; Wolcott, 1993). Spotts (1999) indicated that if instructors are expected to use instructional technologies—including distance education technologies—they need technical support and training. Therefore, the instructional designers and technology specialists, who provide distance education support and training for faculty, need to be knowledgeable about not only the latest technology, but also the educational methods to use that technology (Irani & Telg, 2001; Telg, 1995).
However, many instructional designers who come from a technology-development background (video producers, computer program developers) do not have theoretical knowledge in instructional design methods as it relates to distance education (Telg, 1995). A study of 14 land-grant universities (Irani & Telg, 2001) found that two thirds (64.3%) of instructional designers who actively worked with faculty to develop distance education courses had had no prior training or knowledge of instructional design methods used in distance education before working at their universities. Twelve of the 14 respondents said they had learned distance education instructional design methods while “on the job.” Telg (1995), in a study of video production specialists who support their universities’ distance education efforts, found that the video production specialists also had learned distance education instructional methods while on the job.
Telg (1996a) recommended that a training curriculum be developed to teach television-production-specialist-turned-instructional-designers the information and skills—particularly knowledge of instructional design—that they needed to perform their jobs, so they can subsequently support faculty members’ efforts. Particular areas of instructional design that video producers said they needed more knowledge in included the following: audience identification and needs, adult education theory, adapting content to the technology, distance education theory, interaction methods in distance education, and evaluation techniques in distance education (Telg, 1996b). Because technology changes so rapidly, instructional designers must be provided means to learn about how to apply these informational technologies in learning environments and about learning theories in distance education. Instructional designers must be adequately prepared in order to assist faculty, so that faculty can effectively teach undergraduate and graduate distance courses.
In response to this need, six universities— the University of Florida, Texas A&M University, Texas Tech University, the University of Idaho, the University of Missouri-St. Louis, and Iowa State University—collaborated on a project titled Roadmap to Effective Distance Education Instructional Design. This project was funded by a U.S. Department of Agriculture Higher Education Challenge Grant to develop effective materials and innovative approaches to better prepare instructional designers at land-grant universities and other universities with agricultural academic programs to support their universities’ distance education teaching programs. This “train-thetrainer” approach provided distance education instructional designers—who may be learning instructional design theory and practice on the job—with skills and knowledge to more effectively help faculty members develop distance education courses.
The Roadmap training program included six “destinations” or modules. A 1-week orientation session was conducted before the program to expose participants to the types of technology used in the destinations. Content was delivered on a monthly basis beginning in September and ending in April, with a 2-month break in December and January. Each destination featured synchronous or asynchronous delivery methods—such as streaming video files, narrated PowerPoint presentations, chats, and threaded discussion boards—to provide participants with various examples of how to deliver educational materials at a distance. In addition, Web-based training materials—PDF files, video clips, and related materials—were provided to the participants to use in the training of their own faculty members.
At the conclusion of each destination, participants were asked to complete a micro-project—a short assignment designed to show that the participants understood and could apply the content that was presented. The microprojects varied, based on the content and competencies of the destination. They had three weeks to complete the content and the microproject. During the fourth week, the destination coordinators evaluated the micro-projects and returned their comments to participants.
The faculty researchers involved in this project wanted to determine if knowledge and skills were acquired by these professionals as a result of the Roadmap training program. Therefore, the purpose of this study was to determine training participants’ growth (learning) in distance education instructional design competencies as a result of the 9-month Web-based training program.
REVIEW OF LITERATURE
Previous studies on the use of self-assessment tools to document growth or learning had been conducted with graduate students through a semester-long course (Dooley & Lindner, 2002) and international professionals during a week-long training program (Dooley, Lindner, Dooley, & Alagaraja, 2004), so this framework served as the guidepost for the design for this study. To operationalize, knowledge is a body of information applied directly to the performance of a given activity. Skill is a present, observable competence to perform a learned psychomotor act. Ability is a present competence to perform an observable behavior or a behavior that results in an observable product. Competencies, therefore, establish the behavior requirements needed to be successful. Buford and Lindner (2002) defined competencies as a group of related knowledge, skills, and abilities that affect a major part of an activity.
Competency models can be used as an assessment tool, a career development tool, and as a behavioral requirement benchmarking tool (Dooley et al., 2004; Dooley & Lindner, 2002; Yeung, Woolcock, & Sullivan, 1996). Professionals have used competency models to “clarify organization-specific competencies to improve human performance and unify individual capabilities with organizational core competencies” (Rothwell & Lindhom, 1999, p. 104). Organizations provide training so that individuals will become more competent and, therefore, more effective in their jobs. It is often assumed that training provides the condition for effective learning; however, “from the individual’s perspective, training cannot be assumed to produce learning, nor that learning is always an integral part of training” (Antona-copoulou, 1999, p. 17).
Transfer of learning and the ability to measure learning outcomes as a result of a training program have become major issues in training and development (Antonacopoulou, 1999; Kellie, 1999; Smith, 1999). “Currently, the individual’s perspective is relatively under-researched, thus much remains unclear about the way individuals perceive the association between training and learning and more significantly whether individuals actually learn from training” (Antonacopoulou, 1999, p. 14).
Often, formative and summative self-assessment instruments are used to determine work roles, outputs, and competencies. This study, therefore, adds to the growing body of literature on using self-assessment instruments to measure perceived growth (learning) in competency-based training programs. It is unique in providing authentication measures to compare self-assessment to assignment completion and verification narrative.
METHODS
This study is grounded in the qualitative research paradigm. The general characteristics of this study reflect those identified by Fraenkel and Wallen (1999) as professionally acceptable and appropriate methods for studying a phenomenon when the natural setting is the direct source of data; data are collected holistically from a participant’s perspective; data are analyzed inductively; and data attempts to capture concern for a participant’s behavior, attitude, reason, or motive.
As with any study, it is important for the researcher to establish internal validity, external validity, reliability, and objectivity. However, in the qualitative paradigm these terms are referred to as credibility, transferability, dependability, and confirmability. Credibility and dependability were established by collecting the participants’ responses directly into a database for analysis and verification. The description of the data provided sufficient detail and/or richness so that interpretations of the data could be transferred to other settings. Confirmability was established by coding the data and moving it to an Excel spreadsheet so that it was easily accessible for an audit trail.
The natural setting and prolonged engagement for the study was a 9-month “train–the-trainer” program delivered through WebCT. During the week-long orientation session, participants were asked to complete a self-assessment instrument to measure perceptions of competence in 12 core distance education instructional design competencies, modified from previous research (Dooley & Lindner, 2002). The core competencies, along with characteristics of each competency, are identified in Table 1. Participants were also asked to complete the self-assessment at the conclusion of the training program. Thirty-five participants completed both the pre- and postassessment.
The self-assessment instrument was created based upon the core competencies and characteristics and was intended to serve as a reflection tool for the participants to measure their growth (learning) in the core competency areas. The instrument uses a stair-step approach, rather than a continuum or Likert scale, to visually represent progression from novice (0) to expert (7). The numbers were intended to measure perceived growth rather than any statistical significance. Averages were calculated to show trends in the data only. Participants were asked to verify their answers with a brief narrative (Figure 1). Responses were coded to ensure confidentiality. The constant comparative method was used for data analysis on the verification narrative (Lincoln & Guba, 1985).
RESULTS
The results of this study were reported in two areas: the average growth in core competencies and the open-ended verification of growth categories. In Destination 1, the average growth for the Adult Learning competency changed from 3.43 to 5.20, and for Understanding Teaching at a Distance, the change was 3.43 to 5.39 (see Table 2). One participant, whose self-described knowledge on the Adult Learning competency was a 3 before the training and a 6 afterward, stated:
I understand that adults have very different expectations and needs in learning environments. Adults expect information that is timely and relates to their situations. I understand that people learn in different ways. For example, some are more visual. I also realize that students have their own reasons for participating and learning in a course. It is important to keep these in mind when designing a course. Finally, I believe that distance learners tend to be students who need a more flexible environment for courses than the traditional classroom. (3)
For Destination 2, participants showed growth in Instructional Design by increasing in competence from a 3.50 to a 4.98. The Course Development competency changed from 3.00 to 5.07. A participant, who showed growth from a 4 in the Instructional Design competency at the beginning to a 6 at the project’s completion, had this to say:
The importance of the needs/objectives is paramount to instructional design. They are the reason for the course’s existence and must be the starting point for effective design. Many questions regarding delivery methods, strategies, and assessment will be answered by consulting the objectives and considering the audience. (33)
Core Competencies and Characteristics
| Core Competencies | Characteristics of Core Competencies |
|---|---|
| Destination 1: Adult learning & understanding teaching at a distance | Learning theory |
| Learning styles | |
| Methods for teaching adults | |
| Motivation | |
| Characteristics of distance learners | |
| Differences between distance and traditional environments | |
| Overview of effective distance teaching practices | |
| Destination 2: Instructional design & course development | Needs assessment |
| Writing objectives | |
| Choosing content and methods | |
| Choosing delivery strategies | |
| Assessment | |
| Roles of a development team | |
| Best practices | |
| Destination 3: Delivery strategies for teaching at a distance & instructional technology resources | Web delivery/learning management systems |
| Equipment/hardware (including video, CD-ROM/DVD) | |
| Communication tools | |
| Software (including course management systems, graphics) | |
| Expertise | |
| Technical support | |
| Funding | |
| Destination 4: Advanced interaction methods & accessibility | Threaded discussions/chats as tools for communication |
| Interactive teaching strategies | |
| Types of disabilities | |
| Legislation | |
| Considerations and options for designers | |
| Destination 5: Planning and conducting evaluation & evaluation analysis and reporting | Purposes |
| Formative and summative evaluation | |
| Evaluation methods (surveys, focus groups, rubrics) | |
| Challenges of collecting data online | |
| Analyzing data | |
| Reporting results | |
| Destination 6: Administrative issues & training and support | Principles of best practices |
| Marketing | |
| Copyright | |
| Training of faculty and students | |
| Student support services |
For Destination 3, participants changed from 3.82 to 5.41 in the Delivery Strategies for Teaching at a Distance competency and 3.93 to 5.17 in the Instructional Technology Resources competency. A respondent, who changed from a 3 to a 6 in Delivery Strategies, wrote that by the end of the program he or she could:
recognize the diversity of delivery strategies that are available and the pluses and minuses associated with each method. I understand the need for adequate technical support to effectively implement and maintain equipment and to use development software. (17)
In Destination 4, participants had the most average growth, from 3.07 to 5.34 in Advanced Interaction Methods. Individuals had the lowest initial score (2.75) in the area of Accessibility, but did improve to a 4.68 by the conclusion of the program. A participant who was a 2 in the Accessibility competency at the beginning stated, “This is my weakest area thus far. I admit this is an area I have not adequately considered or applied” (15). By the end the same respondent claimed, “I did not know the tools or sites to use to evaluate accessibility. I thoroughly enjoyed this component and shared the sites with the Web folks in my College and University.”
The Planning and Conducting Evaluation competency showed the second-highest growth, with averages ranging from a 2.93 at the beginning to a 5.12 by the end. Evaluation Analysis and Reporting had a competence recorded initially at 3.25 and final competence at 4.76. An intriguing verification of a respondent who was a 3 in the Evaluation Analysis and Reporting competency at the beginning of the program, but by the end measured as a 6, felt that “our (institution’s) current evaluation data is not analyzed in a critical way” (27). The respondent wrote:
Self-Reported Before and After Scores on the Project’s Core Competencies
| Gender | ||||||
|---|---|---|---|---|---|---|
| Overall Average | Male | Female | ||||
| Core Competency | Before | After | Before | After | Before | After |
| Adult learning | 3.43 | 5.20 | 3.42 | 4.95 | 3.44 | 5.43 |
| Understanding teaching at a distance | 3.43 | 5.39 | 3.42 | 5.00 | 3.44 | 5.76 |
| Instructional design | 3.50 | 4.98 | 3.75 | 4.75 | 3.31 | 5.19 |
| Course development | 3.00 | 5.07 | 2.75 | 4.80 | 3.19 | 5.33 |
| Delivery strategies for teaching at a distance | 3.82 | 5.41 | 3.58 | 5.20 | 4.00 | 5.62 |
| Instructional technology resources | 3.93 | 5.17 | 3.50 | 4.90 | 4.25 | 5.43 |
| Advanced interaction methods | 3.07 | 5.34 | 2.92 | 4.80 | 3.19 | 5.86 |
| Accessibility | 2.75 | 4.68 | 2.25 | 4.20 | 3.12 | 5.14 |
| Planning and conducting evaluation | 2.93 | 5.12 | 2.92 | 4.85 | 2.94 | 5.38 |
| Evaluation analysis and reporting | 3.25 | 4.76 | 3.83 | 4.75 | 2.81 | 4.76 |
| Administrative issues | 2.79 | 4.85 | 2.67 | 4.40 | 2.88 | 5.29 |
| Training and support | 4.07 | 5.17 | 3.92 | 4.75 | 4.19 | 5.57 |
Answers provided on a 7-point scale, with 1 = “novice” and 7 = “expert.”
This section gave insight into the use of survey tools, interviews, questionnaires, and focus groups. Methods for assessing and evaluating course assignments, the use of rubrics, peer review, and faculty assessment. We also got to use that fun “flashlight” tool (Flashlight Online). (27)
For Destination 6, participants rated their competence at a 2.79 at the beginning and a 4.85 at the end for Administrative Issues. In the area of Training and Support, respondents were already fairly high in competence at 4.07 and showed the smallest growth, to 5.17. A participant, measuring a 2 in Training and Support initially and a 5 at the conclusion, provided this comment:
The host of new and unique methods of content delivery surely required a closer look at marketing and copyright issues and an increase in the value of developing and following “best practices.” The workload of instructional designers must be balanced by proactive approaches to training and supporting the new “user-friendly” technologies. Student and faculty orientation to technologies can ensure consistent and proper use and allow the focus to remain on the course content…where it should be. (33)
The collaborators on this project conducted a 6-month follow-up to determine competency retention and transfer of skills/knowledge gained as a result of this project. There were 51 responses to the summative assessment. Many respondents reported using copies of the materials as handouts to give to faculty and in workshops. Additionally, several respondents said they used the information in writing proposals for grants or new courses. They also reported using the information in designing their own courses or courses for other faculty members, as well as in teaching. One respondent said, “I have tried to take a look at my adult learners and be more sensitive to their needs, skill levels, and technology fears.” Respondents noted that they had used what they learned in Roadmap in their jobs (M = 4.01, SD = 1.00, on a 5-point scale where “1 = strongly disagree” and “5 = strongly agree”), the content was useful (M = 4.19, SD = .44), and the content related to their professional work (M = 4.34, SD = .77). In addition, participants indicated that they would be able to perform basic principles, based on the core competencies presented in Roadmap.
CONCLUSIONS AND IMPLICATIONS
An identified need existed for this type of project to address the issue of providing adequate support for people developing distance education courses (Campus Computing Survey, 1999). At the end of the Roadmap to Effective Distance Education Instructional Design project, most participants commented they had an increase in competency levels through their participation in the program, which was the overall goal of Roadmap. The respondents perceived their competence to improve with an average change of 1.77, or close to 2 incremental steps, in the “novice” to “expert” range.
There is a continued need to develop and refine assessment instruments to evaluate and authenticate learning. The findings of this study contribute to the growing body of literature related to identifying and assessing competencies. This model has been used in both a 15-week semester with graduate students and with international professionals in a one-week format with similar results (Dooley et al., 2004; Dooley & Lindner, 2002). The researchers were concerned that because of the duration of this project, the retention and transfer of knowledge, skills, and abilities would diminish. That was not the case with this study. The model used in this study provided a tool to measure competence in instructional design for distance education.
Content has to be applicable and engaging to the target audience, and should be focused on the learner’s environment. This was the plan for Roadmap. The content had to be focused on the professional’s workplace needs. Participants had to demonstrate with the microprojects that they could apply what they learned, so they had to put their learning into practice. The researchers recommend a similar approach for those developing competency-based measures for professionals in other areas, especially for those designing instructional/learning environments.
AUTHOR AND PROJECT CONTRIBUTOR INFORMATION:
This article is based on a collaborative effort of all of the authors and contributors listed. This project was funded by a U.S. Department of Agriculture Higher Education Challenge Grant (#2002-38411-12091).
Project Contributors
Becky Raulerson, project coordinator, University of Florida
Nick Place, associate professor, University of Florida
Colleen Swain, associate professor, University of Florida
Cheryl Bielema, instructional development specialist, University of Missouri-St. Louis
Erik Anderson, director of agricultural communications, University of Idaho
Allan Schmidt, instructional technologist, Iowa State University
Richard Carter, professor, Iowa State University
Cindy Akers, associate professor, Texas Tech University
Chad Davis, assistant professor, Texas Tech University
