This paper presents the results of a systematic effort to collect student feedback on a Web-based distance education course offered by a major university in the midwestern United States. The findings of this study show that some students experienced what Hara and Kling (2000) call distance education student “distress” and that, while they found the course to be a satisfactory educational experience overall, students were “distressed” by technological challenges, ambiguous instructions, and their online interactions with the instructor.
Introduction
Even given the various educational reform movements and changes in pedagogical theory that have come and gone in the past decades, almost all postsecondary students are still educated in a primarily lecture-based educational system (Nunn, 1996). The communication patterns and characteristics of these face-to-face lecture-based environments, whether or not they are the best or most educationally effective environments, are very familiar to students. The shift to an online learning environment can be quite disruptive and challenging to students used to a typical classroom environment (Boehle, 2000; Gale, 2000; Rangecroft et al., 1999). Almost every aspect of the educational process changes in this new environment: class discussions are replaced with e-mail messages and bulletin board postings; paper handouts are replaced by Web pages, and papers are uploaded to the professor rather than being slipped under his or her door.
A rapidly increasing number of postsecondary students are taking distance education courses every year—over 1.5 million in the 1997-1998 school year, according to a National Center for Education Statistics (NCES, 2000) study—and online courses are the major growth area in distance education, with 82% of the institutions in the NCES study planning to start or increase their online offerings. However, currently there is very little solid research to identify key issues related to student satisfaction with online distance education courses. We hope that the results of this current study will help to close this gap in the research.
Purpose of the study
In this study, the researchers explored students’ satisfaction with the instructor, activities, and materials; whether or not the students felt that they met their learning goals; their estimations of their potential to transfer what they had learned in the course to other settings; and the overall impact of the course in their lives, whether it was worth the effort or not. The choice of these areas of inquiry was based upon Kirkpatrick’s (1998) four levels of evaluation, the standard method of measuring the impact of training in the corporate world and higher education settings. However, this study, as it is based solely upon student self-reported data, can only directly measure level 1 (student reaction) and was not designed to fully measure Kirkpatrick Levels 2 through 4 (student learning, transfer, and impact).
The researchers compared their findings to those obtained by Hara and Kling (2000) in a similar study based on the student evaluation of an online course, and used their definition of “distress” to discuss students’ negative feedback regarding their online course experience. It is hoped that this evaluation effort will help other online instructors and course designers in their efforts to design online courses that will be satisfying to their students.
Literature review
The term “distance education” or “distance learning” refers to the teaching-learning arrangement in which the learner and teacher are separated by location and/or time (Moore, 1990). Throughout the past few decades, the field of distance education has taken advantage of current technologies, incorporating into the teaching and learning environment telecommunication technologies such as radio and television broadcasting, audio and video recording, and two-way interactive audio and video.
In the last ten years, there has been a huge growth in the use of synchronous and asynchronous computer-based interaction tools on the Internet or the World Wide Web (Web) (Moore & Kearsley, 1996). The Web is a distributed, hypermedia-based, platform-independent architecture for sharing information. Web-based education is defined as education delivered in whole or in part using the Web and related technologies (Khan, 1997). There are many other similar terms used to describe Web-based education, such as: online courseware, learnware, online distance education, etc. Today, Internet-based distance learning is the most rapidly growing aspect of education and training in the world. According to a study done by the U.S. Department of Education’s National Center for Educational Statistics (1998), more than 79% of the nation’s public, four-year institutions were offering these types of courses.
While this new format of education is replacing, at least partially, the traditional on-campus experience for many students, there is no satisfying answer for the question of how this technology can best be used to approximate (or better) the traditional on-campus classroom environment. As stated by Windschitl (1998) and Moore (1990), much of the published work about distance education has been anecdotal descriptions of activities such as setting up on-line mentoring programs or how to get students involved in collaborative Web-based activities with other schools. There is not much solid research on the evaluation of Internet-based distance education (Keegan, 1996).
Perhaps in part due to this lack of information, much attention was recently given, (including an article in the New York Times online edition) to a study by Hara and Kling (2000) that focused on the frustration expressed by students in an online distance education course, an extreme level of frustration that Hara and Kling refer to as “distress.” Certainly, the results of the Hara and Kling study should concern distance educators and administrators at institutions that offer online courses. But as the New York Times article stated, “The report looks at just one class—and a small one at that—so it is not a survey of distance education courses as a whole, and few if any generalizations can be drawn from it” (Mendels, 2000, p. 1).
Another recent study, by Graham, et al. (2000), evaluated a series of online courses, and painted a much more positive view of the online education experience. According to the Graham et al. study, major critical success factors for ensuring student satisfaction in an online setting are: encouraging active learning, encouraging student-faculty contact, encouraging cooperation between students, and giving prompt feedback. However, further studies looking at the experiences of online distance education students need to be performed before any serious judgments of the pros and cons of this new educational delivery system are made.
The present study will, we hope, add to the body of knowledge about how students experience this new type of college course.
Definition of “Distress”
Hara and Kling (2000) defined distance education ‘distress’ as “situations that the students…find particularly troublesome.” They found five main causes of distress for the distance students in the graduate-level online course they evaluated:
social isolation
overwhelming e-mail communication
lack of instructor feedback
technical problems
ambiguous instructions
Similarly, even given their overall positive review of their courses, Graham et al. (2000) found isolation (“students did not really interact to any large degree in any of the courses” [p. 21]), overwhelming email communication (though this was cited by instructors, not students), and problems with instructor feedback. Technical problems and ambiguous instructions, however, were not cited as causes of distress by the Graham et al. students. In this study, we will evaluate a graduate-level course and see if the students report the same causes of distress.
Research methods
This study focuses on student evaluations of an online, graduate-level course offered by the School of Education at a large midwestern state university. Similar to the course analyzed by Hara and Kling (2000), the course was an educational technology course in which students learned how to utilize information technologies in their areas of subject-matter expertise. It was taught utilizing an extensive Website developed by the instructor. The course Website contained the course syllabus and schedule, the course readings (which were all available online), grading rubrics, samples of student work from past semesters, links to relevant online resources and a gateway to the class discussion area, which utilized an asynchronous conferencing tool. The class size was not large: eleven students registered for the course that semester. The instructor for the course under study was a native English speaker, and had previous experience as an online educator, having taught this course and one other course through online means in previous semesters. Students were located in Indiana, Hawaii, Iowa, Tennessee, North Dakota, and Sweden.
Because the online course would not have class meetings in a physical location and students were so geographically dispersed, it was impossible for the researchers to gather observation data from class sessions and in-person interviews with students. Therefore, the researchers designed Web-based questionnaires that would allow them to collect data via the Internet.
The researchers utilized a eight-step process for developing these questionnaires:
reviewed the literature in this area;
interviewed instructor as part of an informal needs analysis;
reviewed course content/activities;
developed questions focused on Kirkpatrick’s Four Levels of Evaluation.
Some questions were chosen from previous course evaluation instruments developed by team members and colleagues and an instrument contained in a related research report (Ngeow & Holder, 1997);
developed online survey instruments;
administered pilot testing of instruments;
revised instruments based on pilot testing; and
integrated surveys into the course Website.
The evaluation instruments were designed as Web forms, which could be filled out by the students while visiting the course Website. The researchers chose to use Web forms for several reasons. First, since the course was Web-based, it seemed most appropriate to present the evaluation forms by this delivery method. Second, email, another Internet-based option, was judged to be more difficult for the novice to reply to. Third, the researchers wanted the students to be able to complete the forms easily and quickly, and the checkboxes and fill-in areas of a Web-based form allowed for rapid completion time. The researchers used a CGI program called Transform to process the forms through a password-protected section of the School of Education Website. Transform sent the student responses to the researchers (but not the instructor) by email. These responses were then entered into a Microsoft Excel spreadsheet to make analysis easier and more efficient.
The researchers informed the students, both in the email messages that requested that the students fill out the Web-based survey instruments, and on the forms themselves, that their answers to the questions would be confidential, and they would not share their comments with the instructor except in summary form without any names attached. While participation in the study was not mandatory, of the eleven students in the course, only two students did not complete the survey instruments—one student dropped the course, and the other became ill during the semester.
The survey instruments were composed of five-level Likert scale (Strongly Agree to Strongly Disagree) questions, ranking questions, and open-ended questions. The researchers decided to use Likert scale questions rather than open-ended questions for three reasons. First, Likert is a common format for course evaluations (and would be thus familiar to the students). Second, it is easier to evaluate Likert scale questions quantitatively. Finally, the researchers believed students were more willing to answer questions that can quickly be responded, and the Likert format allows for faster responses than open-ended or fill-in-the-blank responses. There were also two ranking questions: one asked students to rank elements of the course based on a least enjoyable to most enjoyable scale, and the other asked students to rank elements of the course based on a least difficult to most difficult rank. The instructor requested that the researchers add an open-ended question at the end of the instrument to capture student response information that might otherwise be lost.
The content of the questions on the survey instruments, as noted above, were based on Kirkpatrick’s Four Levels of Evaluation (Kirkpatrick, 1998), even though only the first level could be directly measured. Each of the Kirkpatrick levels focused on a different aspect of student response to the course, and so it was decided, in consultation with the instructor, that basing questions on the four levels would be the best way to ensure that the fullest range of student response to the course instructor, materials and activities was measured. The researchers’ definitions of the four levels follow:
Level 1 (Student satisfaction)
Students’ feelings of satisfaction regarding the instructor, methods and content, including:
course materials;
course activities;
instructor performance;
overall rating for course.
Level 2 (Student learning)
Students’ perception of their gain in knowledge and skills related to the content of the course.
Level 3 (Transfer of learning)
Students’ conception of how they would apply the knowledge and skills gained in the course to their job/school environment.
Level 4 (Cost/benefit impact)
Students’ estimations of the efficiency and effectiveness of the course and its impact on student’s performance at school/work.
As was previously mentioned, it is important to note that for all of these levels, the researchers were only able to measure the students’ perception and estimation of the degree to which the course satisfied their requirements for each level. In particular, for Levels 3 and 4 (Transfer of Learning and Impact), the researchers could only measure the students’ opinions regarding whether or not they would be able to transfer their learning to other settings and the impact of that transfer; they were not able to actually measure whether or not this transfer occurred, or would occur.
Results of the study
In this section of the paper, we present the results of the student course evaluation. The following paragraphs describe the most significant findings of our study. Overall, the student responses can be summarized as follows, including both positive and negative feedback:
Student Feedback
Level 1
Overall, students gave positive comments regarding this course. However, many students were not very satisfied with their interactions with the instructor. Also, a small number of students were not well prepared for the technological requirements of the course, which caused them frustration. One survey respondent described his/her ‘distress’ over the way the course was organized: “I am totally frustrated. I absolutely do not know how this class is organized and how to access the information I need. I hate Long Distance education and I never plan to do this ever again. It has made me rethink even using the Internet in my class at school. I hate this. I hate this. I hate this."
Level 2
The students reported a moderate level of learning. Most students were very confident that they had accomplished the learning objectives of the course.
Level 3
Students expected professional benefits in the future from taking the course.
Level 4
Students responded that the course cost them more money than on-campus course, but saved them time. The majority of the students felt that the cost/benefit ratio of the course was very favorable. However, some students cited decreased time to spend with their family and on their work as an opportunity cost of the course. One student stated as part of his/her survey response, “I have little time for my family. (…) Much time is spent with the mechanics of the course, which includes posting on work on Websites and submitting it.” Another student said, “I have been forced to reduce some of my time at work.”
Our study echoed the findings of both Hara and Kling (2000) and Graham et al. (2000) in that the instructor’s online communication practices were a probable cause of student ‘distress.’ In this course, many students were not very satisfied with their online interactions with the instructor. While the students recognized the instructor’s knowledge of the subject matter, the majority of the students (5 of 9 respondents) did not give positive evaluations of their online interactions with the instructor.
Another source of difficulty for the students in this study, which was also found by Hara and Kling (2000), related to using the technology required for the course. The course required students to navigate through various Websites, send and respond to email messages, and use a Web-based asynchronous conferencing tool for class discussion purposes. Some students reported difficulties with utilizing the technologies required for these tasks. A few students complained about the way the course Website, which was developed by the instructor, was structured. Graham et al. (2000) also commented on the need for improving the organization and presentation of information on the course Websites that they evaluated.
Overall, in comparing our findings with those of Hara and Kling, our students reported that they shared 3 of 5 causes of student “distress” with the students in the Hara and Kling study:
social isolation (not found)
overwhelming email communication (not found)
lack of instructor feedback (found)
technical problems (found)
ambiguous instructions (found)
Thus, the majority of the causes of “distress” were the same for both courses. This, combined with the findings of the Graham et al. study, suggests that this definition of the causes of student distress in an online educational environment may have some validity.
Recommendations
After a careful review of the student responses to the course evaluation instruments, and our analysis of the causes of distance education student “distress,” we have developed the following recommendations for online course instructors:
Instructors should review their practices in responding to students’ email and Web-based conferencing posts, to ensure that they are providing sufficient and appropriate feedback.
Instructors should specify the technological requirements in the syllabus, and arrange for technical support.
Instructors should conduct a usability test on the course requirements and other instructions to ensure that they are clear and non-ambiguous.
Limitations
The research design for this study faced a number of limitations, chief among them being a limited time frame for administering the evaluation. Other limitations were:
the small sample size (although it was the entire class);
the lack of in-person or telephone interviews with students;
the possibility that education students may be atypical, compared to all postsecondary students;
the lack of follow-up short-term (immediately post-course) and long-term (six months afterwards) evaluation and;
as stated before, the lack of ability to directly measure Kirkpatrick’s Level 2, 3 and 4 (student learning, transfer of learning, and impact).
Conclusions
Of course, as with the Hara and Kling (2000) study, the generalizability of the data in this survey is necessarily limited because it is only based on the student evaluations of a single course. However, the researchers offer these findings as part of a larger effort to understand the experiences of an ever-growing number of students as they experience a new format for education. Even though these students may never set foot on campus or step into an instructor’s office, the researchers feel that it is important that their voices are heard as distance education instructors and administrators continue to develop online courses and programs.
Based on the experiences of the students in this online course, the researchers have two primary recommendations for others developing such courses. First, to ensure that the students feel “the pains of innovation” (Hara & Kling, 2000) to the least degree possible, the instructor should clearly specify the technological requirements at the outset of the course and should supply, or arrange for others to supply, the necessary technical support. Second, more attention also needs to be paid to developing and refining methods of moderating and monitoring online discussions with students. The researchers suggest that instructors should review their practices regarding responding to students’ email and web-based conferencing posts, to ensure that they are providing sufficient, appropriate, and reasonably prompt feedback.
The authors would like to thank Yun-Man Chang and Hsiao-Ming Lee for their assistance in the early stages of this study.
