Comparison of Instructors' and Students' Perceptions of the Effectiveness of Online Courses

The number of online courses has dramatically increased in the United States in recent years. The largest increases have occurred in associate degree institutions, where 72% agree that online courses are part of their long-term strategy, as compared to 58% in 2003 (Allen & Seaman, 2005). Thus, Web-based pedagogy is a critical factor that affects today's education in community colleges.

As technology has improved, the delivery of online courses has changed significantly. For example, more media and interactivity, as well as community publishing are now part of the delivery of course content. But as Web-based pedagogy brings new opportunities, it also brings new challenges to both instructors and students. For example, in 2000 the BellSouth Foundation (2003) launched the “Power to Teach Program” to explore ways to help create a critical mass of K-12 teachers capable of incorporating technology into everyday classroom experiences. The data showed that while teachers felt they were making dramatic leaps in using technology to create new learning experiences for their students, students saw few changes in their classroom instruction. In fact, students revealed that they were hungry for more opportunities to use technology in the learning environment.

So, how can there be such a disparity in perceptions? The Web-based learning model—the Model of Community of Inquiry—developed at the University of Alberta, assumes that learning occurs within the community through the interaction of three elements: cognitive presence, social presence, and teaching presence (Garrison, Anderson, & Archer, 2000; Seok, 2006, 2007a, 2007b). One of the contributing factors may have to do with teaching presence. While cognitive and social presences are considered core elements in learning, whether or not learning is achieved depends on the presence of a teacher to facilitate the learning activities. Other contributing factors may be at play as well with regard to social presence.

Although the BellSouth program focused on K-12, in general, it sheds light on the importance of perceptions in the classroom. For instructors to deliver more effective Web-based learning in community college settings, we need to know more about the perceptions of online courses by the producers and consumers of online technology—instructors and students.

This study compared instructors' and students' perceptions of the effectiveness of online courses in community college settings. Course effectiveness was analyzed along the following composites: flexibility, user interface, navigation, getting started, technical assistance, course management, universal design, communications, online instructional design, and content. This study was an extension of a validation study conducted by Soek (2006) that piloted a comprehensive online course evaluation instrument at the postsecondary level.

In this study, two survey questionnaires were used to gather students' and instructors' demographic information and their perceptions of online course effectiveness in community colleges. Variables of gender, age, native language, academic major, educational level, technology skill, and experience with online courses were taken into consideration to determine if they significantly affected subjects' perceptions of the effectiveness of Web-based courses. Educational theories, principles, practices, and research on perceptions of course effectiveness in Web-based instruction was the research base for this study. The findings may be used to help instructors and course designers gain a better understanding of how to evaluate, design, and deliver more effective web-based learning environments for each subscale and across students' demographics (Seok, 2007a, 2007b, 2008).

In order to compare instructors' and students' perceptions of the effectiveness of online courses in a community college setting, answers were sought to the following research questions:

• Research Question 1: Are there significant relationships between students' and instructors' perceptions of online course effectiveness and students' and instructors' demographic characteristics (i.e., gender, age, native language, major, educational level, technology skill, and learning experience)?

• Research Question 2: Are there significant differences between students' perceptions and instructors' perceptions of online course effectiveness?

To better understand the importance of the perceptions of online course effectiveness, the subsequent review has been organized into the following sections: the pedagogical features, principles, and practices of online learning; students' perceptions of online course effectiveness; and instructors' perceptions of online course effectiveness.

The World Wide Web provides new functionality in transmitting information and has enhanced the ability to educate students electronically in both stand-alone tutorials and online workshops. As such, the Web has increased opportunities for learning and alternative formats for information delivery (Dwyer, Barbieri, & Doerr, 1995).

The pedagogical features of major course management systems include collaboration and communication, content creation and delivery, administrative tools, learning tools, and assessment tools (Online Learning, 2005). The five categories that address the major components of an educational event as designed and developed for distance education include: learning goals and content presentation, interactions, assessment and measurement, instructional media and tools, and learner support systems and services (Ragan, 1999).

In summary, the pedagogical features of online learning include: a tool for learning, interactions (communication and collaboration), enhanced content and its delivery, measurement, and technical and administrative services. The goals of these features are intended to enhance interactions between learners and peer learners, learners and content, learners and technology, and learners and instructors (Seok, 2007a, 2007b, 2008).

All in all, students are no longer totally dependent on instructors for learning when in an online learning environment when there is easy access to course content and information resources. For online learning to be successful, instructors as well as students must take on new roles in the teaching-learning relationship. Instructors must be willing to motivate and release control of learning to students (Illinois Online Network, 2007) and students must be able to assume more independence.

Findings from studies examining perceptions of online course effectiveness have been positive. For example, O'Malley and McGraw (1999) found that distance learning and online learning technologies were perceived by students as having some benefits, although they were not necessarily knowledge related. Their study investigated student perceptions of the effectiveness of distance and online learning to determine which dimensions of online learning provide advantages relative to traditional methodologies. A 128-item 7-point Likert-type survey questionnaire was developed and administered to students in a variety of business courses at a participating university. Survey results indicated that students appeared to be ambiguous regarding the effectiveness of online learning when compared with traditional methodologies. Most of the relative advantages of online learning were related to saving time, scheduling, and being able to take more courses.

Koohang and Durante (2003) designed a 10-item Likert-type instrument based on instructional objectives to collect information about students' perceptions of web-based distance learning activities and the assignment portion of a hybrid program. Subjects were 106 students enrolled in an undergraduate hybrid program on management fundamentals designed for working adults. Results indicated that, overall, students greatly perceived that the web-based distance learning activities/assignments portion of their program promoted learning. A significant difference was found among levels of learners' experience with the Internet and their perceptions of the web-based distance learning activities/assignments portion of the hybrid program. In other words, subjects who had more experience with the Internet indicated significantly more favorable perceptions of the web-based distance learning activities and assignment portion than did subjects with less experience with technology and the Internet.

While finally, Jurczyk, Benson, and Savery (2004) used a standards-based approach to measure student perceptions in web-based courses in an effort to develop a process for evaluating perceptions. Standards were based on a literature review and interviews with 147 individuals. Subjects included faculty members, students, and administrators at six leading accredited institutions in distance education. Forty-five benchmarks were identified and organized into seven categories: institutional support, course development, teaching/learning process, course structure, evaluation and assessment, student support, and faculty support. In addition to a process, a questionnaire was developed to measure student attitudes before, during, and after taking the web-based course.

Positive findings have also been found with regard to instructors' perceptions of online course effectiveness. For example, Wingard (2004) found that a significant number of faculty thought that adding web-improved preparations, for themselves as well as for their students, contributed to greater student engagement and active learning in the classroom. Within an online environment, faculty often felt they were more familiar with their students' academic progress during the term and reported a growing expectation that students could take more responsibility for independently learning the fundamentals from the readily available resources provided on the Web.

Guidera (2004) investigated the perceptions of faculty at both public and nonprofit private institutions in the United States—including 2year institutions, 4-year colleges, and universities—on the effectiveness of online instruction in terms of the seven principles of effective undergraduate education. These seven principles include good instructional practice, encourage student-faculty contact, encourage cooperation among students, encourage active learning, provide prompt feedback, emphasize time on task, communicate high expectations, and respect diverse talents and ways of learning. The research results indicated that online instruction was rated slightly more effective overall and more effective for promoting prompt feedback, time on task, respect for diverse learning styles, and communicating high expectations. However, it was rated less effective for promoting student-faculty contact and cooperation among students. Interestingly, the perceived effectiveness was higher for experienced faculty, which increased with the number of online courses taught (Guidera, 2004).

Finally, Seok (2006, 2007a, 2007b, 2008) identified indicators to evaluate online instruction at the postsecondary level. Ninety-nine indicators applicable to the evaluation of online instruction were identified and validated. In doing so, Seok (2006) developed 11 categories as follows: flexibility (e.g., schedules, technical skills, and work settings), user interface (e.g., consistent user interface and appealing screens), navigation (e.g., menus and labels), getting started (e.g., class orientation and performance expectations), technical assistance (e.g., online help and on-call support), course management—instructor (e.g., manage student assignments and monitor student progress), course management—student (e.g., detailed syllabus and benchmarks for completing course requirements), universal design (e.g., accommodate students with disabilities and allow students to vary font size), communication (e.g., individual responses, distribute announcements, discussions, and chat), online instructional design (e.g., glossary of all terms, content maps of all lessons, and additional resources for enrichment), and content. These validated indicators can be transformed into item scales and subscales to evaluate the effectiveness of learning instruction. It is these indicators that are examined in this study. It is hoped that instruments derived from this study's findings may be used to evaluate productivity and processes of e-learning (Seok, 2007a, 2007b).

This study is a composite of findings from quantitative statistics—multidimensional scaling and a one-way analysis of variance (ANOVA) from two large-scale studies—“Perceptions of Students and Instructors of Online and Web-Enhanced Course Effectiveness in Community Colleges” (Tung, 2007) and “Validation of Indicators by Rating the Proximity Between Similarity and Dissimilarity among Indicators in Pairs for Online Course Evaluation in Postsecondary Education” (Seok, 2006).

Two groups participated in this study—instructors teaching community-college-level online courses and students enrolled in community-college-level online courses. Using convenience sampling, 281 instructors and 176 students were recruited via e-mail and asked to complete online survey questionnaires. A total of 193 instructors and 141 students completed the survey of online course effectiveness.

There were eight independent variables (see Appendix) for instructors and students: gender, age, native language, academic major, educational level, technology skills, and number of online courses completed (for students) and taught (for instructors).

Eleven dependent variables were included to ascertain students' perceptions of online course effectiveness: flexibility, user interface, navigation, getting started, technical assistance, course management (instructor), course management (student), universal design, communications, instructional design, and content.

Two online survey questionnaires based on the Inventory for Online Course Evaluation in Postsecondary Education (Seok, 2006) were administered to collect instructors' and students' perceptions of the effectiveness of various aspects of online learning and teaching. Two different formats—one for instructors and one of students—consisted of the same questions. Each questionnaire consisted of two sections: perceptions of course effectiveness and personal data.

The first section included 99 questions and used a 5-point Likert-type scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree). Subjects were asked to indicate whether they agreed or disagreed with each statement, which were all positively worded. An open-ended question was included at the end of survey to collect additional comments about course effectiveness in teaching and learning. The second section asked for students' and instructors' gender, age, native language, academic major, educational level, technology skills, number of online courses completed (for students) and taught (for instructor).

Survey questionnaires focused on subjects' perceptions in the following areas of online instruction: flexibility, user interface, navigation, getting started, technical assistance, course management (instructor), course management (student), universal design, communications, instructional design, and content.

Validity. Ninety-nine items were developed and solidly validated in a previous study (Seok, 2006) to develop an instrument to evaluate online courses by utilizing 4, 856 pairwise comparisons by subject matter experts (SMEs) by implementing multidimensional scaling (MDS).

Reliability. Cronbach's coefficient alphas were used to compute internal consistency estimates of reliability for each subscale (flexibility, user interface, navigation, getting started, technical assistance, course management, universal design, communications, instructional design, and content) of the measurement instrument.

According to Nunnaly (1978), 0.7 is an acceptable reliability coefficient. Using an internal consistency estimate of reliability, individuals are administered a measure with multiple parts on a single occasion (Green & Salkind, 2005). In the current instrument, no items needed to be reverse-scaled since all survey questions presented positively worded statements. Furthermore, all items shared the same metric, since the response scale for all items ranged from 1 = strongly disagree to 5 = strongly agree. All subscales in this study were found to have alpha levels greater than 0.7, indicating acceptable reliability (see Table 1).

In this study, factor analysis consisted of the four typical phases: Phase I—item development, Phase II—item distribution to large sampling, Phase III—factor analysis application, and Phase IV—item revision.

As mentioned earlier, MDS was conducted. This quantitative statistic application is the process of item validation, which corresponds to Phase I in factor analysis. Multidimensional scaling utilizes SMEs to validate the identified and developed items (which is one of the differences between factor analysis and multidimensional scaling) and for the purpose of this study, consists of six phases.

Phase I involved a literature search to identify indicators crucial to effective online learning. Phase II involved two subject matter experts (SMEs) reaching consensus on the indicators, consisting of a comprehensive set of 99 independent indicators considered to be representative of online course structure for postsecondary education. Phase III involved developing an online MDS instrument to accommodate the 4, 851 pairwise comparisons in random order. Phase IV involved a panel of four SME judges rating the 4, 851 pairwise comparisons ([99*98]/2), where the number 99 indicates that there were 99 items developed. Their ratings of the similarity of indicators confirmed the existence of the indicators' validity. A multiple-point scale from 1 to 9 was used, with higher numbers indicating similarity and lower numbers indicating dissimilarity. And finally, phase V involved data collection via an online MDS instrument. Phase VI involved applying MDS analysis to the comparisons. A survey was distributed to a larger sample for this study to compare the perceptions of instructors and students corresponding to Phase II of factor analysis. A one-way ANOVA was conducted after MDS and after the items were solidly validated (Seok, 2006, 2007a, 2007b, 2008).

The results of the Soek (2006) study indicated that a three-dimensional solution was an appropriate model for analysis, thus indicating a .24222 of the fit values of STRESS and 74% of R². Three dimensions were labeled as accessibility, adaptability, and clarity of communication. Four clusters of indicators were identified as contextual accommodation, instructional access, guided learning, and organizational clarity (Seok, 2006, 2007a, 2007b). An ANOVA test was used to measure whether:

• the means on subjects' perceptions of course effectiveness subscales were significantly different across gender, age, native language, academic majors, educational levels, and technology skills; and

• if the means on students' and instructors' perceptions of course effectiveness were significantly different.

The results relating to student perceptions of course effectiveness suggested satisfactory evidence with internal consistency estimates ranging from .76 to .96 for the individual subscales within the surveys. Instructor perceptions of course effectiveness also suggested satisfactory evidence, with internal consistency estimates ranging from .72 to .92 for the individual subscales within the surveys. Using Cronbach's coefficient alphas, the subscale scores were found to be reasonably reliable for this ANOVA study (see Table 1). The descriptive results (see Table 2) indicated that, overall, students and instructors had positive perceptions of the effectiveness of online courses.

The subsequent sections describe the data collected on the demographic section of the surveys, which showed significant findings.

Gender was found to be a statistically significant factor for instructors and students alike. Both female instructors and students had statistically significant higher perceptions of the online course effectiveness than males. One-way ANOVAs indicated a significant difference between male and female instructors' perceptions of the subscales as follows. For getting started, F(2, 191) = 4.97, p < .05, the mean of female instructors was significantly higher (M = 4.2, SD = .57) than the mean of male instructors (M = 4.1, SD = .61). In the case of technical assistance, F(2, 191) = 7.83, p < .05, the mean of female instructors was significantly higher (M = 3.9, SD = .75) than the mean of male instructors (M = 3.6, SD = .80). And with regard to universal design, F(2, 191) = 7.77, p < .05, the mean of female instructors was significantly higher (M = 3.8, SD = .59) than the mean of male instructors (M = 3.6, SD = .64).

One-way ANOVAs also indicated a significant difference between male and female students' perceptions in six subscales, including user interface, getting started, technical assistance, communications, online instructional design, and content. The largest gaps between means were found in three subscales—user interface, online instructional design, and content. For user interface communications, F(2, 139) = 7.12, p < .05, the mean of female students was significantly higher (M = 4.0, SD = .53) than the mean of male students (M = 3.7, SD = .62). In the case of online instructional design, F(2, 139) = 6.50, p < .05, the mean of female students was significantly higher (M = 4.0, SD = .62) than the mean of male students (M = 3.6, SD = .61). And with regard to content, F(2, 139) =13.26, p < .05, the mean of female students was significantly higher (M = 4.2, SD = .61) than the mean of the male group (M = 3.7, SD = .72).

For instructors' teaching experience, Table 3 shows that small positive correlations were found between online course taught and most subscales, except flexibility and navigation. Thus, increased online teaching experience contributed to instructors' perceptions of delivering more effectively designed online courses. These findings indicate that the delivery of effective courses may depend upon increased teaching experience.

For students' native language, the results indicated that native-English-speaking students had statistically significant higher perceptions of online course effectiveness than non-native-English-speaking students in the user interface and getting started subscales. For user interface, F(2, 139) = 5.11, p < .05, the mean of English speakers was significantly higher (M = 3.9, SD = .54) than the mean of the other language speaker (M = 3.4, SD = .76), while with respect to getting started, F(2, 139) = 5.39, p < .05, the mean of English speakers was significantly higher (M = 4.0, SD = .62) than the mean of other language speakers (M = 3.4, SD = .88). Non-native-English-speaking students had significantly lower satisfaction with online course effectiveness in the user interface and getting started subscales than native English-speaking students. Such a finding deserves further attention in future research. Topics might include how computer user interface design affects students' online learning with regard to cultural diversity and how to provide a better getting-started design in online learning for non-native-English-speaking students of diverse cultural backgrounds. No statistically significant differences were found in instructors' perceptions of online course effectiveness subscales across instructors' native languages and academic majors.

Significant differences were found across instructors' education levels (less than high school, associate's degree, bachelor's degree, master's degree, and doctoral degree) in their perceptions of the following online course effectiveness subscales: navigation, F(5, 188) = 2.66, p < .05; getting started, F(5, 188) = 4.32, p < .05; course management, F(5, 188) = 3.22, p < .05; and universal design, F(5, 188) = 3.64, p < .05.

Significant differences also emerged across students' educational levels in their perceptions of the online course effectiveness in the subscales of instructional design, F(6, 135) = 3.68, p < .05, and content F(6, 135) = 5.20, p < .05. This finding indicates that instructors with higher educational levels may deliver more effectively designed online courses in terms of navigation, getting started, course management, and universal design.

Results indicated significant differences across instructors' technology skills in their perceptions of the online course effectiveness of the following subscales: flexibility, F(3, 190) = 3.81, p < .05; user interface, F(3, 190) = 3.82, p < .05; communications, F(3, 190) = 4.44, p < .05; online instructional design, F(3, 190) = 4.55, p < .05; and content, F(3, 190) = 6.74, p < .05. For the students, statistically significant differences were found across technology skills in their perceptions of the online course effectiveness for the content subscale, F(3, 138) = 3.83, p < .05.

The flexibility mean value for beginners was higher than the value for instructors with intermediate skills. However, both mean values were significantly lower than the mean value for instructors with advanced technology skills. Thus, increased technology skills may contribute to instructors' perceptions of delivering more effectively designed online courses in the following course effectiveness subscales: flexibility, user interface, communications, online instructional design, and content. As a result, the delivery of effective online courses may depend upon increased technology skills as illustrated in Table 4.

Results of the ANOVA showed that instructors had statistically significant higher perceptions toward online course effectiveness than students in the following subscales: getting started, course management, communications, and content (see Table 1).

The aforementioned results indicated that instructors had statistically significant higher perceptions of the effectiveness of online courses than did students. These results deserve further attention and should be subsequently researched. For example, it would be advantageous to explore whether the difference in perceptions are due to generational gaps between what Prensky (2003) refers to as digital natives and digital immigrants.

Results also showed positive correlations between online courses taught and most subscales except flexibility and navigation. These findings indicated that the delivery of effective courses depends upon teaching experience. Instructors with advanced technology skills had statistically significant higher perceptions in some course effectiveness subscales, further implying that, with rapid advances of information technology, popular use of course management systems, comfort levels of using a computer, and increased online learning/teaching experience, come the increased positive perceptions of online course effectiveness on the part of both students and instructors.

Furthermore, native-English-speaking students had statistically significant higher perceptions of online course effectiveness than did non-native-English-speaking students in terms of user interface and getting started. Non-native-English-speaking students had significantly lower satisfaction with online course effectiveness in these subscales than did native-English-speakers. Such findings may imply that non-native-English-speakers may need additional assistance when taking online courses.

Based on the findings of this study, further research is recommended related to the following course effectiveness subscales: flexibility, communications, and online instructional design. For flexibility, further research is recommended relating to effective self-paced online learning and teaching because of the demands of self-paced flexibility from online course students and concerns expressed by faculty about students' success in the self-paced learning environment. For communications, it is recommended that appropriate communication theories and principles that meet the needs of online learning environments are included in future research. While finally, for online instructional design, it is recommended that future studies address students' learning preferences (Dunn & Dunn, 1978) in online instructional design. It is also the recommendation of the authors that non-native-English speakers' perceptions see further attention in future research.

The descriptive results indicated that, overall, students and instructors had positive perceptions of online course effectiveness. Findings, generally speaking, are in line with past studies investigating the perceptions of instructors and students with regard to online courses.

As previously discussed, there are three major aspects of online learning: cognitive, social, and teaching (Seok, 2006, 2007a, 2007b). The findings of this research related to the teaching aspects in terms of students with different language backgrounds other than English, teaching experience, and technology skills. Teaching experience and technology skills were found to be highly correlated with online course effectiveness, while students with different language backgrounds other than English had low perceptions of online course effectiveness.

Taken together, these findings underscore the importance of faculty development in online learning and the development of cognitive and social strategies for students with different cultural and linguistic backgrounds.

Nominal variable for the independent variables: Gender: 1= female and 2 = male; Native language: 1= English and 2 = other; Academic major: 1 = business and 2 = continued education, 3 = engineering or technology, 4 = fine arts, 5 = humanities, 6 = information technology, 7 = mathematics, 8 = natural sciences, 9 = nursing, dental or allied health, and 10 = social sciences.

Ordinal variables for the independent variables: Educational level: 1 = less than high school, 2 = high school diploma or GED, 3 = associate's degree, 4 = bachelor's degree, 5 = master's degree, and 6 = doctoral degree; Technology skills: 1 = beginner: students who have minimum experience in using a computer and the Internet, 2 = intermediate: students who use a computer and the Internet on a daily basis, and 3 = advanced: students who have the ability to solve their problems in using a computer and the Internet.

Interval variable: Number of online courses taught and completed.