Evaluation of the Effect of Multisite Distance Education on Knowledge Gained in a Plant Propagation Course Free

Effective instructors use a variety of formal and informal methods to evaluate the effectiveness of the course learning experience for their students (Willis, 2002). Some of these methods focus on formal assessment of student learning, such as use of quizzes, tests, examinations, and assignments. However, teachers are also accustomed to drawing on their own perceptions as to how students are reacting to the material, whether they understand what is being said, how well they are able to model or replicate a process being taught, and so on.

In the distance education setting, instructors face many additional changes that introduce a measure of complexity to the course evaluation process. These challenges can include difficulties inherent in teaching in an unfamiliar environment to students from a wide variety of backgrounds and with little to no face-to-face interaction, which limits the instructor's ability to observe student reactions in the classroom (Willis, 2002). Further, the introduction of a technological delivery method creates a need to consider how the use of technology influences the learning experience. Research has shown that student experiences with distance education technologies are related to their attitudes and perceptions of course satisfaction and perceived learning (Howland & Moore, 2002; Biner, 1993), and that students in “hybrid” distance learning courses with onand off-site components may differ in their perceptions of a course (Hackman & Walker, 1995).

The development of strategies to reach students who are place-bound due to their job, families, or community responsibilities is an important opportunity for land grant colleges and universities, in particular. Indeed, land-grant universities and many other agricultural institutions have been at the forefront in developing extensive infrastructures to facilitate distance education delivery of courses to a diverse community of learners, both traditional and non-traditional (Miller & Pilcher, 1999). Most of these programs involve technological delivery of distance education coursework in a variety of majors at both the graduate and undergraduate levels utilizing teleconferencing, videotape, and the Internet. In a study of distance education delivery methods and evaluation practices of higher education institutions that deliver agriculturally related content, Roberts, Irani, Lundy, and Telg (2004) found that most of the agricultural institutions they surveyed were using a mix of hybrid delivery methods, including Web-based course management software and video conferencing. From a course student evaluation standpoint, the most common method for developing a distance education evaluation instrument was to make a few revisions to the on-campus instrument already utilized at each institution.

Although not specifically part of a course evaluation process, many researchers have used assessment instruments to measure students' attitudes about such topics as the overall distance education experience, technology used, instructional methods, and interaction techniques (Biner, 1993; Diebel, McInnis, & Edge, 1998). In an early study of students taking an interactive television course, Sorensen found that students' primary complaint was poor reception (video and audio), based on technological constraints. Gray and Miller (1999) found that age appeared to be an attitudinal factor, relating to desired interaction levels in distance education courses distributed by videotape and an interactive video network. Older students placed a higher value on learner-content interaction and learner-interface interaction than did younger students.

Others have examined the impact of learning styles on academic performance. Marrison and Frick (1994) reported that field-independent learners found multimedia instruction easier and more exciting, but found no significant differences in achievement between fielddependent and field-independent learners. Miller (1997) compared students' learning styles with their attitudes about distance learning courses. He found that field-independent learners were more positive about the possibility of taking another course by videotape or interactive video. Rudd and Telg (1998) found that student learning style did not have a significant impact on student performance. However, there was a significant difference in student performance between the on-site class section and the off-site section of the course.

One of the most well-known attempts to develop course evaluation instrumentation for distance education courses was conducted by Biner (1993), who used empirical data to develop the Telecourse Evaluation Questionnaire (TEQ). This instrument consisted of the four constructs of: Instruction/Instructor Characteristics, Technological Characteristics, Course Management and Coordination, and General and Demographic Information. Thirty-four Likert-type questions were used to address the first three constructs. The General and Demographics construct consisted of an additional eight questions. Biner conducted several studies using the TEQ to assess student perceptions and satisfaction in the telecourse setting (Biner et al., 1997; Biner et al., 1994). Subsequent to the introduction of web based distance education platforms, the TEQ has been adapted to assess these same factors in an online course environment (Ricketts, Irani, & Jones, 2003).

In fall, 2002, a plant propagation course that had previously been taught traditionally, in a live classroom format, was restructured for distance education videoconference and Web-based delivery to six on- and off-campus sites through the College of Agricultural and Life Sciences at a land-grant institution in the Southern U.S. Instruction originated from two off-campus sites, something that had not been done previously, and was delivered via compressed video conferencing network to participating sites around the state. The course was developed and coordinated by a multisite teaching team that included two coordinator/instructors, site facilitators, and local laboratory instructors for each site.

Given the complexity of the project, the goals of the instructional team were to develop a framework for the development and coordination of the course that would maximize available technologies to deliver course content, and provide opportunities for learnerinstructor and learner-learner interactions. Therefore, the instructional development included a combination of compressed video, Web, Flash animation, and CD. WebCT was utilized to provide course management and interaction capabilities via discussion forums, chat, and e-mail.

Since the course design and development were innovative in nature and fairly resource intensive in terms of time and production costs, the decision was made to include a formal pretest posttest evaluation of the course, in an attempt to ascertain the effectiveness of the course in terms of student knowledge gain, as well as provide feedback on instructional techniques, delivery methods, and student expectations and satisfaction. Based on the above, the objectives of this study were to:

1. Determine if students significantly gained in perceived knowledge as a result of taking the course.

2. Describe student perceptions of the course, instructor and delivery method, as well as their expectations of and satisfaction with the course learning experience.

3. Describe the on and off site student population in terms of their overall evaluation of the instructor and the course in which they were enrolled.

The development of satellite programs was designed to provide the opportunity for place-bound students to earn a baccalaureate degree in several different disciplines while allowing for enrollment expansion outside the boundaries of the main campus in Gainesville. The development of the plant propagation course was an attempt by the university's Department of Environmental Horticulture to embrace this vision for bigger and broader technology-based educational programs. Adoption of this strategy enabled a single faculty member with expertise in a given subject to deliver a course and share his or her knowledge statewide. Currently, four Environmental Horticulture courses are offered via interactive videoconferencing, with several more scheduled for future delivery. This alleviates the necessity to have multiple instructors with expertise in these areas of instruction and serves as a mechanism to standardize course material statewide.

The course was initially delivered in the fall semester of 2002, to 65 students at six sites, including the main campus. The course was dual listed for both undergraduate and graduate enrollment. The two course instructors were located at two of the remote sites, and conducted lecture from these sites, which was broadcast to the other sites, including the main campus site. This was a first for the CALS distance education program.

Objectives for the course were to:

1. Enable students to acquire a comprehensive knowledge of the science of plant propagation, including the effects of plant physiological reactions, anatomical structures, and environmental influences on material use in plant propagation.

2. Develop a level of skill in the art of plant propagation by seeds and vegetative organs.

3. Stimulate critical thinking through class discussions, outside reading assignments, outside projects and field practice.

4. Develop a vocabulary of plant propagation terminology and its proper use orally and in writing.

5. Encourage an interest, understanding and appreciation of the principles and techniques of plant propagation.

Organization of the course content was based on dividing the material into six lecture modules comprised of classroom video conference lectures (75% of course), guest lectures and demonstrations using streamlining video, narrated PowerPoint and animation, nursery tours using streamlining video, and discussion via e-mail and discussion boards in WebCT (25% of course). The lecture modules included general aspects of propagation; seed propagation; vegetative propagation by cuttings; grafting and budding; layering, propagation of specialized stems and roots, clonal selection; and cell and tissue culture.

Each lecture module was broken down into a series of Web-based lectures, video presentations accessed from the CD, and discussion/interaction opportunities conducted through WebCT. In addition, a special animation of the life cycle of angiosperms was developed with a view toward helping students better conceptualize this detailed process. The animation was produced using Adobe Flash animation software, and included illustration of the basic life cycle, a series of “popup” animations to provide greater detail at certain stages of the life cycle, and an animated bee that initiates the process of fertilization. Music and voice narration were linked to each step of the life cycle to personalize the animation and explain key components of the life cycle process (Wilson & Thetford, 2003).

Because courses delivered by distance education create a physical distance between students and the instructor, incorporation of planned interactive exercises is of crucial importance. Also students from main and remote, off-site campuses may have dramatically different perceptions and expectations of a course. Although often done as a purely economic measure to boost enrollment and Full Time Equivalency (FTE) for distance delivered courses, some studies indicate that, in these types of combined “near and far” distance learning experiences, differences in perceptions between on- and off-site students do exist. These differences are seen especially in terms of students' perceptions of the level of interaction and the amount of feedback offered to the remote site students (Hackman & Walker, 1995; Willis, 1992).

As an instructor, it takes very specific teaching skills to meet the needs and interests of such a diverse population of students; to customize lectures based on students' various levels of knowledge and to maintain enthusiasm and motivation in the classroom. Distance education courses, by their nature, incorporate instructional and technological components of instruction at a more synergistic level than traditional, live classes. This creates challenges in terms of evaluation of faculty, and the distance education courses that they teach, since the technology component adds complexity to standard course evaluation techniques. As discussed earlier, challenges with technology unfamiliarity, glitches, and clarity can play into a student's evaluation of the course and are typically not addressed in course evaluations. Teaching can also be more challenging with the instructor's limited ability to observe student reactions in the classroom (Willis, 2002).

To address these issues, an instrument was developed with the help of the course instructor that included 10 knowledge items designed to assess perceived knowledge change/gain in plant propagation concepts that were taught in the course through a pre- and post-test assessment. The resulting questionnaire was administered via the Web course management system used in the course to students (N = 53) during the first week of class. Items utilized a Likert-type scale, with responses ranging from 1 (strongly disagree) to 5 (strongly agree). Post hoc alpha reliability for the scale was r = .81. For the posttest, conducted during the last week of class, the 10 knowledge items were again used. A set of 46 questions (see Figure 1) adapted from Biner's TEQ were designed and implemented at this stage to evaluate the instructor, the technologies, and teaching techniques used and course management and support services. In addition, two demographic items measuring gender and university classification were used. Item scales ranged from 1 (very poor) to 5 (very good). Post hoc reliability for the scaled items was alpha = .96. (Since the course was team taught, students evaluated each instructor separately. For the purposes of this study, the pre- and post-test was conducted solely on content taught by and the performance of the lead instructor.)

A total of 44 students completed both the pretest and posttest instruments. Of these, 64% (n = 28) were undergraduates, 9% were graduate students (n = 4) and 27% (n = 12) were nondegree seeking students. Forty-one percent of student respondents were male (n = 18) and 59% (n = 26) were female.

Means were calculated for both pre- and post-test responses to the ten content knowledge items and then compared using paired sample t tests. Results showed that, on every item, students perceived a significant content knowledge gain from the pretest to the posttest assessment. The greatest gain in perceived content knowledge was in plant clone variation, followed by micropropagation, then grafting and budding (see Table 1).

In addition to the content knowledge items, the posttest included a set of items adapted from the TEQ that were designed to evaluate aspects of the instructor's performance, the course technology, management, and support services. In addition, a series of items was used to assess students' perceptions of the course experience, workload, whether or not they would take another distance education course, whether or not they would have been able to take the course if it had not been offered via distance, and how many prior distance learning courses they had taken. Means were calculated for all items and are displayed in Table 2.

Finally, to assess the effect of location on students' perceptions of the instructor and the course overall, two items drawn from the existing standard course evaluation instrument were utilized. In the official course evaluation instrument, students were asked to rate first the instructor and then the course on a scale ranging from 1 (poor) to 5 (excellent). Reported summated means were obtained from the instructor for each site involved in the course. Inspection of these means showed that the highest means were from site 4, one of the remote sites from which the instructors delivered the course, while the lowest means were

from the main campus site (site 7). Table 3 displays these means by course site.

The advent of technology based distance education has created new challenges with respect to the course evaluation process. Given that such courses may include diverse elements such as multiple delivery technologies, remote sites, and use of facilitators and co-instructors, researchers have questioned whether course evaluations should be specific to the individual course situation, separate from traditional class evaluations but otherwise generic, or should they share the same evaluation form as traditional, live classes. As was the case at this university, many institutions' course evaluation instruments do not evaluate technology, other than a series of items focused on laboratory instruction. Further, most do not include questions to assess important distance education concepts such as level of interaction, support, and/or facilitation.

This study was designed as a comprehensive description and evaluation of a plant propagation course developed specifically for multisite distance education delivery and team-taught by a team of instructors. As such, it utilized an evaluation instrument that enabled student respondents to evaluate all aspects of the course, the instructor and the technology used to deliver it. In general, based on the findings, the course was successful in achieving several of its major goals, including developing students' perceived knowledge and skill in the area of plant propagation, as well as “encouraging interest, understanding and appreciation” for the principles and techniques utilized in this field.

Specifically, in order to assess perceived knowledge, students were asked to self-rate their knowledge on each of ten content areas covered in the course, both at the beginning of the course and then again at the end. Results showed statistically significant differences in perceived knowledge from the pretest to the posttest, with increases in student respondents' ratings of every item. In addition, students evaluated the major elements of the course, including the technology, the instructor, and the management of the course, favorably. The instructor's enthusiasm and preparation received especially strong ratings, as did as the accessibility and helpfulness of the site facilitators. Further, almost half of the students in this study indicated that they would not have been able to take the course if it had not been offered in a distance education format, a finding that makes sense, given the use of multiple remote sites to deliver the course throughout the state.

Since this was a course delivered to and taught from remote sites, the role of site facilitators was obviously an important consideration for most students taking the course. Interestingly, however, although both the course and the instructor received very favorable overall ratings, the highest ratings came from one of the off-campus sites where the lead instructor was teaching from, while the lowest evaluation came for the main campus site, suggesting that proximity to the instructor was an important factor for students. This supports the literature, which suggests that on-campus students differ from remote site distance learners in that they tend to be more negative about their course experiences in a distance setting (Hackman & Walker, 1995).

In general, when asked if they would take another distance education course like the one they had just taken, over 70 % of students in this study said they would, a finding that suggests that students had an overall favorable impression of the course, its instructor, and the teaching method employed. By this measure, it could be inferred that the instructors were successful in their efforts to stimulate student interests and engage them in the subject matter of the course, despite geographic distance.

In conclusion, the plant propagation course evaluated in this study used a wide range of technologies, teaching methods, sites, and even more than one instructor, all of which represented a complex challenge for both conducting the class and attempting to evaluate it. By utilizing the adapted TEQ questionnaire form, student respondents were able to separately evaluate the instructor, the technology, and the management of the course, including such aspects as the library and the registrar's office—a somewhat more holistic approach to evaluating a distance education course experience than the more standard type of course evaluation. This approach seemed to be effective, and it seems likely that a similar approach should be considered when attempting to evaluate distance courses with multiple delivery methods and/or delivery sites.