Student Perceptions of Quality and Satisfaction in Online Education Free

The growth of online education has encouraged institutions to search for ways to ensure quality in online course offerings (Chua & Lam, 2007; Dringus, 2000; Puzziferro & Shelton, 2008; Ross, Batzer, & Bennington 2002). Puzziferro and Shelton (2008) suggested that online course development needs a “common framework for consistency, design, pedagogy and content” (p. 119). The framework used in this study was the Quality Matters (QM) framework, which is a faculty-centered, peer review process for ensuring quality in online and hybrid courses (Quality Matters, 2010). The QM rubric is continuously updated and revised based on peer review and current research (Legon, 2009). This framework is built upon eight broad standards: course overview and introduction, learning objectives (competencies), assessment and measurement, instructional materials, learner interaction and engagement, course technology, learner support, and accessibility.

The purpose of the study was to examine student perceptions of quality and satisfaction in regionally accredited online courses. It is intended to support online education accreditation standards for peer reviewed online courses. The study intended to answer the following questions: (1) Is there a significant difference in levels of student satisfaction between online courses that have undergone a systematic faculty peer review process and online courses that have not undergone a systematic faculty peer review process?; (2) Is there a significant difference in levels of student satisfaction between online courses that have not undergone a systematic faculty peer review process but are affiliated with a peer review program and online courses that have no affiliation with a peer review program?; (3) Which factors of quality instruction most directly relate to increased levels of student satisfaction in online courses that have been faculty peer reviewed?; and (4) Which factors of quality instruction most directly relate to increased levels of student satisfaction in online courses that have not been faculty peer reviewed but are affiliated with a peer review program?

Researchers agree that accreditation is viewed as a “seal of approval” (Casey, 2008, p. 49) and a way to govern that “an institution or program meets threshold quality criteria” (Dill, Massey, Williams, & Cook, 1996). The review of literature identified several frameworks for validating quality in an online course. The quality frameworks considered for this study were: Best Practices for Electronically Offered Degree and Certification Programs, Sloan Consortium, Quality Matters, and Universitas 21 Global.

Quality Matters was chosen as the framework for this study in an effort to add to the scarce literature currently available regarding the framework. According to Puzziferro and Shelton (2008), QM has received “national recognition for their research-based rubric and interinstitutional peer review processes” (p. 125). The QM rubric was initially created using The Guiding Principles for Distance Learning and Distance Learning Evaluation Guide as a reference (Shattuck, 2010). This rubric also included the Best Practices for Electronically Offered Degree and Certificate Programs in its development, which is the set of standards implemented by the eight regionally accrediting bodies in the United States (Legon, 2006). QM differs from traditional accreditation processes in that it focuses on evaluating the quality of individual online courses, rather than evaluating overall programs and institutions (Legon, 2006). The QM rubric includes eight broad standards: course overview and introduction, learning objectives (competencies), assessment and measurement, instructional materials, learner interaction and engagement, course technology, learner support, and accessibility. Those standards are further broken down into 41 specific standards that are assigned a rating of 1, 2, or 3, where 3 = essential, 2 = very important, and 1 = important. To meet the minimum requirements of the rubric, all standards deemed essential must be included in an online course. Additionally, out of 95 possible rubric points, an online course must obtain a minimum of 85% (or 81 points total) to meet the course design level of acceptance. A formal peer review by QM includes three QM-certified reviewers. Of these reviewers, one must be from an institution other than the one being reviewed and one must be a content specialist (Legon & Runyon, 2007; MarylandOnline, 2010). Quality Matters also offers a subscription to institutions choosing not to undergo a formal peer review. This subscription gives institutions access to the QM online tools, trainings, and materials without a formal peer review.

Ralston-Berg and Nath (2008) completed a study to determine how much students valued those QM rubric standards in their online course. For the purposes of this study, value means “how much they wanted the feature in an online course” (Ralston-Berg & Nath, 2008, p. 2). Results of the study showed that students ranked 40 of 41 standards as either very important or essential, with netiquette being the only standard considered not valuable. This study also conducted correlations between student perceived satisfaction and each of the QM standards. The correlations showed that students who perceived their satisfaction as high were correlated with those who ranked the QM standards are more valuable and vice versa. Artino (2011) backed up these findings with a study that explored students’ perceived quality of an online course that was designed with the QM framework. Artino's study showed that students viewed almost all of the QM standards as valuable.

Swan, Matthews, Bogle, Boles, and Day (2011) conducted a study that combined both the QM and community of inquiry (CoI) frameworks in a redesign of an online course. The purpose of the study was to redesign a course to meet QM level of acceptance and then tweak it based on the subjective and constructivist CoI ratings. Findings of the study indicated that using the two frameworks simultaneously may improve student outcomes.

Legon (2006) compared the “Best Practices for Electronically Offered Degree and Certificate Programs”—which is used by the eight regionally accrediting agencies in the United States—with the QM rubric to see if an online course designed from the QM framework met accreditation guidelines for online education. Legon found that the QM rubric was consistent with accreditation standards for online education and he recommended the rubric could be used as a record of accreditation compliance.

Little (2009) conducted a study using 2005 QM rubric and the College of Public Heath (COPH) Online Course Standards. The purpose of the study was to compare the evaluation of online courses between the two frameworks. Results indicated that both frameworks provided accurate evaluations of online courses, so users should choose the framework that is most user-friendly. The QM rubric was chosen in this study as being the more user-friendly of the two.

Aman (2009) conducted a study to determine whether courses that were faculty peer reviewed would lead to higher student satisfaction and retention rates in online courses. This study found that courses that had been peer reviewed were consistent with higher student satisfaction of the course. However, there were no significant findings in regards to student retention.

This study included upper-level and/or graduate online courses offered at senior institutions accredited by the Southern Association of Colleges and Schools (SACS). Participants were divided into three categories:

1. Recognized—online course has undergone a formal peer review through QM;

2. Subscribed—online course has not undergone a formal peer review through QM but the institution is subscribed to QM tools, trainings, and materials; and

3. Nonaffiliated—online course has not undergone a formal peer review through QM and the institution is not subscribed to QM.

Institutions identified in one category were excluded from the other two categories.

Aman's Satisfaction Questionnaire (2009) was the instrument used in student data collection. This questionnaire included only five of the QM standards: learning objectives, assessment and measurement, instructional materials, learner interaction and engagement, and course technology. This questionnaire included a 5-point Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree) with a neutral midpoint. This questionnaire asked 30 questions. A few questions were asked with regard to student demographics that could be indirectly related to student satisfaction with an online course: comfort with technology, number of online courses completed, gender and age. General background information about the course was also collected: course introduction, technology support, and student support. The remainder of the survey asked questions regarding each of the QM standards included in the questionnaire in an attempt to establish the importance of each standard to the student and the student's satisfaction with each standard in their online course. For the purpose of this questionnaire, importance is defined as “a general, student reported, benchmark for the value placed on each of the factors,” and satisfaction is defined as “the student's level of approval for that specific factor with the online course” (Aman, 2009, p. 75). The questionnaire concluded with one final question to assess the student's overall satisfaction with their online course. Construct validity of the questionnaire was obtained through the review of literature and content validity was obtained through a panel of online learning experts and QM experts. Reliability of the questionnaire was obtained with a mean Cronbach's Alpha of .84.

The researcher developed a Qualitative Course Design Survey that was sent to either the QM institutional representative (if applicable) or the course designer of each course included in this study. Institutions in the “recognized” category were asked questions regarding the level of QM implementation across the institution. Institutions in the “subscribed” category were asked questions regarding how many of the QM resources were used in the design of the course and which specific tools were used. Institutions in the nonaffiliated category were asked questions regarding whether the institution was using another type of peer review evaluation (other than QM).

Of the 334 students solicited in the recognized category, 22 chose to participate in the student satisfaction questionnaire. Of the 764 students solicited in the subscribed category, 48 chose to participate in the student satisfaction questionnaire. Of the 676 students solicited in the nonaffiliated category, 87 chose to participate in the student satisfaction questionnaire. This comes to a total of 157 responses out of 1,774 solicited, which is a student response rate of 9%. Additionally, one QM institutional representative (out of four solicited) in the recognized category chose to participate in the course designer survey. Three course designers (out of 26 solicited) in the subscribed category chose to participate in the course designer survey. Three course designers (out of 24 solicited) in the nonaffiliated category chose to participate in the course designer survey. This comes to a total of 7 responses out of 54 solicited, which is a course designer response rate of 13%.

The Southern Association of Colleges and Schools Commission on Colleges Member, Candidate, and Applicant List (2011) showed 805 institutions that were regionally accredited through SACS. A comparison of this list, the QM website, and the individual websites of each senior institution included in these two references revealed 10 institutions that met criteria for the recognized category, 72 institutions that met criteria for the subscribed category, and 453 institutions that met criteria for the nonaffiliated category. There were 270 institutions that did not meet research criteria to be included in this study. Research Randomizer (http://www.randomizer.org/) randomly selected 10 institutions from each category that were contacted to request participation in this study. Survey data were collected via Survey-Monkey to ensure anonymity of the respondents.

From the recognized category, 4 institutions agreed to participate, 3 declined participation, and 3 did not respond. Those who declined gave the following reasons: (1) institution is undergoing a restructure, (2) solicitation is prohibited at the institution, and (3) another study is already in progress. A total of 334 students across the 4 participating institutions were found to be enrolled in the 8 online courses that had been formally reviewed by QM. These students were e-mailed a research invitation asking for their participation in an online qualitative satisfaction questionnaire. Follow-up requests were sent anywhere from one to four times (depending on the length of the course), in 2-week intervals between April 26, 2012 and July 30, 2012.

From the subscribed category, 3 institutions agreed to participate, 2 declined participation and 5 did not respond. Those who declined gave the following reasons: (1) solicitation is prohibited at the institution and (2) institution is undergoing QM implementation and is not ready to be part of a study. A total of 764 students across the 3 participating institutions were enrolled in the 26 online courses chosen from this institution to be included in the study. Online courses chosen from this institution were similar to the courses studied from other participating institutions. These students were e-mailed a research invitation asking for their participation in an online qualitative satisfaction questionnaire. Follow-up requests were sent anywhere from one to four times (depending on the length of the course), in 2-week intervals between April 26, 2012 and July 30, 2012.

From the nonaffiliated category, 3 institutions agreed to participate, 3 declined participation and 4 did not respond. Those who declined gave the following reasons: (1) another study is already in progress and (2) solicitation is prohibited at the institution. A total of 676 students across the 3 participating institutions were enrolled in the 24 online courses chosen from this institution to be included in the study. Online courses chosen from this institution were similar to the courses studied from other participating institutions. These students were e-mailed a research invitation asking for their participation in an online qualitative satisfaction questionnaire. Follow-up requests were sent anywhere from one to four times (depending on the length of the course), in 2-week intervals between April 26, 2012 and July 30, 2012.

Additionally, the QM institutional representative (if applicable) or the course designer for the online course was e-mailed a research invitation requesting their participation in an online qualitative survey to determine how they use peer review in the design of their online courses. Follow-up requests were sent anywhere from one to four times, in two-week intervals between April 26, 2012 and July 30, 2012.

Statistics showed that the mean age of participants was 33.87, with a gender split between 107 female respondents and 49 male respondents (one respondent's gender was unknown). A one-way ANOVA determined that overall satisfaction was reportedly higher in the age groups 25 to 35 years (M = 4.21, SD = .925) and 55 years or older (M = 4.20, SD = .447) than in the remaining age groups: less than 25 years (M = 3.73, SD = 1.202), 35 to 45 years (M = 3.86, SD = 1.357), 45 to 55 years (M = 3.75, SD = 1.372), and not answered (M = 3.29, SD = 1.704).

The differences between these means were non-significant at the .05 level (p = .326). A one-way ANOVA determined that overall satisfaction was slightly higher in females (M = 3.98, SD = 1.258) than in males (M = 3.75, SD = 1.102). These results were also non-significant at the .05 level (p = .300)

The reported comfort level with technology was varied, with a mean of 3.54 (where one equals very uncomfortable and five equals very comfortable). A one-way ANOVA determined that overall satisfaction was higher in students who were very comfortable and comfortable with distance learning (M = 4.06, SD = 1.131) than in students who were neutral, uncomfortable, or very uncomfortable with distance learning (M = 3.58, SD = 1.323). These results were found significant at the .05 level (p = .026).

The number of completed online courses per respondent was also varied between 0 courses to 42 courses, with a mean of 10.68. A one-way ANOVA determined that overall satisfaction was marginally higher in students who had taken 10 or more previous online courses (M = 3.94, SD = 1.282) than in students who had taken fewer than 10 online courses previously (M = 3.88, SD = 1.150). These results were nonsignificant at the .05 level (p = .744).

Descriptive statistics gauged background information about the online course, such as whether there was a course introduction, technological support, and student support. Most participants reported that the online course in which they were enrolled began with a clear course introduction (M = 4.19, SD = .956). Additionally, most participants reported that technological support for the online features of the course was available (M = 3.96, SD = .992) and that student support for the online course was available (M = 3.89, SD = 1.025).

The first research question, “Is there a significant difference in levels of student satisfaction between online courses that have undergone a systematic faculty peer review process and online courses that have not undergone a systematic faculty peer review process?” determined if the satisfaction mean scores of the recognized group was different from the subscribed and nonaffiliated groups. A two-tailed independent sample t test found that overall satisfaction from the peerreviewed group was reportedly higher (M = 4.14, SD = .910) than overall satisfaction from the two groups that had not been peer reviewed (M = 3.87, SD = 1.256). Assuming equal variances, the difference between the means was non-significant at the .05 level (p = .337).

Because the t test for overall satisfaction was determined to be nonsignificant, the researcher took the mean of each of the five QM factors included in this study and performed an additional t-test analysis on the mean of those QM factor means between the recognized group and the subscribed and nonaffiliated groups. A two-tailed independent sample t test showed the mean of QM factor means for the recognized group was higher (M = 4.2440, SD = .56130) than the mean of QM factor means for the subscribed and nonaffiliated groups (M = 4.1266, SD = .75574). Equal variances assumed, the difference between these two means was found nonsignificant at the .05 level (p = .509). A simple linear regression was not significant (p = .337), indicating that peer review cannot be used to predict overall student satisfaction.

The second research question, “Is there a significant difference in levels of student satisfaction between online courses that have not undergone a systematic faculty peer review process but are affiliated with a peer review program and online courses that have no affiliation with a peer review program?” determined if the satisfaction mean scores of the subscribed group was different from the nonaffiliated group. A two-tailed independent sample t test found that overall satisfaction from the subscribed group was actually lower (M = 3.73, SD = 1.245) than overall satisfaction from the “non-affiliated group (M = 3.94, SD = 1.264). Assuming equal variances, the difference between the means was nonsignificant at the .05 level (p = .399).

Because the t test for overall satisfaction was determined to be nonsignificant, the researcher took the mean of each of the five QM factors included in this study and performed an additional t-test analysis on the mean of those QM factor means between the subscribed group and the nonaffiliated group. A two-tailed independent sample t test showed the mean of QM factor means for the subscribed group was lower (M = 4.1085, SD = .61886) than the mean of QM factor means for the nonaffiliated group (M = 4.1369, SD = .82758). Equal variances assumed, the difference between these two means was found nonsignificant at the .05 level (p = .849). A simple linear regression was not significant (p = .399), indicating that subscribing to peer review tools, trainings, and materials cannot be used to predict overall student satisfaction.

The third research question, “Which factors of quality instruction most directly relate to increased levels of student satisfaction in online courses that have been faculty peer reviewed?” determined if there was a relationship between peer review and student selfreported satisfaction with each of the five QM factors included in this study: learning outcomes, assessment and measurement, learning resources and materials, learner interactions, and online course technology. A Pearson correlation coefficient determined that all correlations were nonsignificant at the .05 level (p = .485, p = .221, p = .415, p = .787, p = .564), suggesting that peer review is not related to student satisfaction with each of the five QM factors.

The fourth research question, “Which factors of quality instruction most directly relate to increased levels of student satisfaction in online courses that have not been faculty peer reviewed but are affiliated with a peer review program?” determined if there was a relationship between being subscribed to peer review tools, trainings, and materials and student selfreported satisfaction with each of the five QM factors included in this study: learning outcomes, assessment and measurement, learning resources and materials, learner interactions, and online course technology. A Pearson correlation coefficient determined that all correlations were nonsignificant at the .05 level (p = .565, p = .866, p = .950, p = .913, p = .997), suggesting that subscribing to peer review tools, trainings, and materials is not related to student satisfaction with each of the five QM factors.

Qualitative data obtained from the “recognized” group indicated that the one institution participating in this study used the QM rubric, faculty development, course development checklist, and QM peer review tools. Qualitative data collected from the “subscribed” group indicated that all three of the participating institutions used the QM rubric for course design, two of those (66.7%) used the course development checklist and one (33.3%) used the faculty development tool. All of these institutions reported using the QM rubric to improve existing courses, while only one reported using it to design new courses. Two of the participating institutions (66.7%) indicated that all courses at their institution were required to use the QM subscription tools, trainings, and materials. Qualitative data collected from the nonaffiliated group indicated that none of the three participating institutions were subscribed to any type of peer review program.

A major limitation of this study was the population and the sample size. The population was limited to only senior institutions accredited by the Southern Association of Colleges and Schools in which upper-level and/or graduate courses were offered in an online format. Ten schools were selected from each category and solicited for participation in this study. Of 1,774 students solicited, only 157 chose to participate, which resulted in a 9% response rate. Additionally, of 54 course designers solicited, only 7 chose to participate, which resulted in a 13% response rate.

The design of the survey was also considered a limitation of this study. Survey results were collected online anonymously, so it was not possible to personalize each participation invitation or to accurately follow-up with nonrespondents. The structure of the survey was also considered a limitation because it asked only closed-ended questions, which did not allow participants to provide supplementary information regarding their online course.

A review of the literature indicated there may be certain standards in online courses that lead to an increase in students’ perceived quality of that course (Ross et al., 2002) and an increase in students’ perceived satisfaction of the course (McGorry, 2003; Sloan Consortium, 2011; Sun, Tsai, Finger, Chen, & Yeh, 2008; Zhao, 2008). This study was built upon the QM framework, which includes 8 broad standards: course overview and introduction, learning objectives, assessment and measurement, instructional materials, learner interaction and engagement, course technology, learner support, and accessibility. To maintain the validity and reliability of the survey instrument used, only five of these standards were assessed in this study: learning outcomes, assessment and measurement, learning resources and materials, learner interactions, and online course technology.

Analyses showed that gender, age, and number of online courses completed were nonsignificant to student perceptions of overall satisfaction in their online course. Richardson and Swan (2003) and Hong (2002) agreed that age was not significant to student perceptions of an online course. Thurmond, Wambach, Connors, and Frey (2002) also confirmed that age is not a predictor of student satisfaction in an online course. Richardson and Swan (2003) found gender to have a positive relationship with student satisfaction in an online course. However, Hong (2002) and Lim (2001) did not find a relationship between gender and student satisfaction in an online course. The literature is mixed regarding the significance of the number of completed online courses to student satisfaction of an online course, with some supporting the significance of the relationship (Arbaugh & Duray, 2002) and some contradicting the significance of the relationship (Richardson & Swan, 2003; Thurmond et al., 2002). Student comfort with distance learning was found to have a significant relationship with student perceptions of overall satisfaction in their online course. This significant finding is supported by the literature (Abdous & Yen, 2010; Lim, 2001; Thurmond et al., 2002).

This study found no significant difference in levels of student satisfaction between online courses that have undergone a systematic faculty peer review process and online courses that have not undergone a faculty peer review process. Further analysis acquired the quality factor means (learning outcomes, assessment and measurement, learning resources and materials, learner interactions, and course technology) and calculated an overall mean, which found higher satisfaction reported from the “recognized” group as opposed to the “subscribed” and nonaffiliated groups. These results, although also nonsignificant, are supported by the literature (Aman, 2009; Artino, 2011; Ralston-Berg & Nath, 2008; Swan et al., 2011). The research further showed that peer review was not a predictor of student satisfaction in an online course. This contradicts the literature review which suggests that undergoing a formal peer review may lead to higher student satisfaction in an online course (Aman, 2009; Swan et al., 2011).

This study found that there is no significant difference in levels of student satisfaction between online courses that have not undergone a systematic faculty peer review process but are affiliated with a peer review program and online courses that have no affiliation. Further analysis acquired the quality factor means (learning outcomes, assessment and measurement, learning resources and materials, learner interactions, and course technology) and calculated an overall mean, which found lower satisfaction reported from the subscribed group as opposed to the nonaffiliated group. These results were also non-significant and contradict the literature (Aman, 2009; Artino, 2011; Ralston-Berg & Nath, 2008; Swan et al., 2011). The research further showed that a subscription to peer review tools, trainings, and materials was not a predictor of student satisfaction in an online course. This contradicts the literature review, which suggests peer review tools, trainings, and materials may be positively related to higher student satisfaction in an online course (Aman, 2009; Artino, 2011; Ralson-Berg & Nath, 2008; Swan et al., 2011).

This study found that peer review was not related to student satisfaction with each of the five QM factors (learning outcomes, assessment and measurement, learning resources and materials, learner interactions, and online course technology). This contradicts the literature review regarding peer review and online course quality (Aman, 2009; Artino, 2011; Ralston-Berg & Nath, 2008).

This study found that a subscription to peer review tools, trainings, and materials was not related to student satisfaction with each of the five QM factors (learning outcomes, assessment and measurement, learning resources and materials, learner interactions, and online course technology). This contradicts the literature review regarding peer review and online course quality (Aman, 2009; Artino, 2011; Ralston-Berg & Nath, 2008; Swan et al., 2011).

Considering the findings of this study, the researcher recommends that a formal peer review be conducted of online classes due to the findings in this study suggesting that a formal peer review may lead to higher student satisfaction in an online course. Although this finding was nonsignificant, the mean overall student satisfaction was still higher in the recognized group than in the subscribed and nonaffiliated groups. This study did not find that having a subscription to peer review tools, trainings, and materials would positively affect student satisfaction in an online course. It is recommended that anyone holding a subscription to a peer review program follow through with a formal peer review to enhance the quality of course design, which may in turn lead to higher student satisfaction with the course.

Considering the findings of this study, the researcher recommends expanding the study to include all QM recognized courses and all institutions that are subscribed to QM, which would drastically increase the population of the study. Expanding the population may lead to a larger sample size. The researcher also recommends using a different approach to solicitation. Wright (2005) suggests that offering an incentive to potential survey participants may increase the participation rate. In order to offer an incentive, however, survey participants will have to be identifiable. Aman (2009) suggested that securing preapproval and acceptance of the study from the faculty teaching the participating online classes may help in increasing the student response rate. Another recommendation is to modify the existing survey instrument to allow openended responses from participants. Finally, the researcher recommends that further research be conducted on the significance of subscribing to peer review tools, trainings, and materials without undergoing a formal peer review.

The researcher's hypothesis was that courses that had undergone a formal peer review, as well as courses that had not undergone a formal peer review but were offered at institutions that were subscribed to peer review tools, trainings, and materials would lead to higher student satisfaction with their online course. A t-test analysis determined that online courses that had undergone a formal peer review did result in higher student satisfaction with the course, although the results were non-significant. However, a t-test analysis also determined that online courses that were offered at institutions that were subscribed to peer review tools, trainings, and materials (without undergoing a formal peer review) led to lower student satisfaction with the course. Those results were also nonsignificant. In both cases, a simple linear regression analysis determined that peer review could not be used to predict student overall satisfaction.

This study also looked at five factors from the QM rubric (learning outcomes, assessment and measurement, learning resources and materials, learner interactions, and online course technology) to determine if there was a relationship with student satisfaction and each of the five factors. A Pearson correlation determined that student satisfaction with each of the five factors was not related to peer review or to a peer review subscription. These results were also found nonsignificant.

A one-way ANOVA determined whether there was a relationship between student satisfaction with their online course and the four demographic variables collected as part of the survey data. The analyses showed that age, gender, and number of online courses completed was not significant to student satisfaction with the online course. However, comfort with distance learning was significant with regards to student satisfaction with their online course. Implications were difficult to project since most of the results were nonsignificant. However, the results did indicate that a formal peer review of an online course may lead to higher student satisfaction with that course. Further research is recommended to validate these findings. It is also recommended that further research be conducted to determine if holding a subscription to peer review tools, trainings, and materials is beneficial to student satisfaction with an online course.