Assessing Faculty Experiences With And Perceptions of An Internal Quality Assurance Process for Undergraduate Distributed Learning Courses: A Pilot Study

Thirty-two percent of students at U.S. colleges and universities are enrolled in at least one online course (Allen & Seaman, 2013). It is no surprise that nearly 70% of chief academic leaders see online learning as critical to their institution’s long-term strategy (Allen & Seaman, 2013). Although national and tional standards in assessing distributed learning (DL) programs, no format-specific standards for quality assurance are required in online higher education courses. Scholars in the field have now begun to collect and suggest internal standards for online course design and delivery (Southard & Mooney, 2015). To ensure the integrity of their online courses and programs, many institutions have voluntarily adopted best practices recognized by these and other leaders in the field of distributed learning. Perhaps most notably, Quality Matters (QM) provides a peer review process designed to certify the quality of online courses (Quality Matters, 2014). Initially funded through a federal grant, QM is an independent organization with 900 subscribing educational institutions.

Leaders in DL education agree that assuring quality online learning in higher education involves institutional vision, commitment, and leadership (Australasian Council, 2014). The University of South Carolina (USC), a public research university with an overall enrollment exceeding 48,000, offers 36 distributed learning-based graduate degrees and certificate programs; seven DL undergraduate degree completion programs; and a wide array of online courses available through other degree programs (Statistical Summary, 2015). USC is accredited through the Southeastern Association of Colleges and Schools Commission on Colleges and follows its Distance and Correspondence Education policies (2014). Leadership at USC has made DL a key academic priority.

The University of South Carolina system is a large, public flagship institution with a RU/ VH Carnegie-classification as a very-high activity Research University. The system consists of four 4-year campuses (USC-Aiken, USC-Beaufort, USC-Columbia, and USCUpstate) that award bachelors and advanced degrees. The system also consists of the Palmetto College regional campuses, which award associate degrees and online bachelor’s degree completion programs (USC-Lancaster, USC-Salkahatchie, USC-Sumter, and USCUnion). The system has an enrollment of 48,166 students (Fall 2014) and 3,354 faculty (Fall 2013). These numbers consist of 39,682 undergraduates, 6,871 graduate students, and 1,613 professional students. At the time of this study, USC had 18,862 online course enrollments, 1,193 courses, 36 online graduate degrees and certificate programs, and seven fully online undergraduate degrees

In 2011, USC’s vice provost assumed leadership of the university’s DL programming. Acknowledging that facilitating learning through technology enhances wider access to a first-rate university education, one of his first priorities was to hire professional instructional designers, whose services would be made available to faculty through the institution’s (CTE). Shortly thereafter, the University became a subscriber to QM, the nationwide peer review system aimed toward ensuring high quality online courses. The QM rubric is a set of standards for course design that reflects nationally recognized, research-based best practices for DL. The CTE now manages the institution’s QM membership, and all DLrelated staff at CTE have completed QM training in applying the QM Rubric, as well as certification to review DL courses for QM standards compliance.

In 2013, the provost announced that, systemwide, all DL courses would undergo an internal quality assurance (QA) review process. Working collaboratively with a faculty committee, the Provost’s Committee on Distributed Learning combined QM criteria with specific guidelines to ensure Americans with Disabilities Act (ADA)-accessibility to create an assessment instrument (“the QA Standards”) for the internal QA process. The provost’s office consistently reinforced the importance of the initiative, and the vice provost personally addressed individual concerns and made certain the process was applied consistently.

In 2014, CTE facilitated the internal QA review process for undergraduate-level DL courses. The process began with department chairs and deans, who identified courses from their areas for review. Many of the courses slated for review had not been recently updated, employed outdated technologies and delivery, and would require substantial revisions to meet the QA standards. Affected faculty worked for several weeks with a designated CTE designer and other support personnel to update their courses and make needed revisions. Once complete, the instructional designer forwarded the course to an appointed faculty committee for final approval.

Recognizing that faculty involvement and buy-in was crucial to success, a key goal was to establish a process that was faculty-driven from start to finish. To ensure that the process was faculty focused, many support initiatives were implemented, including instructional design support. Throughout the course review and revision process, CTE designers intently focused on helping faculty find flexible solutions for meeting the QA Standards—there was no single required design or approach required. Rather, designers integrated the faculty member’s course goals, teaching style, and needs into course building. Additionally, the provost’s office provided faculty whose courses successfully completed the review a $1,500 grant for time, materials, or services (filming, transcription, etc.) required by the course revisions.

In all, 107 courses were identified for QA review the first year. While 29 were retired by their respective departments prior to review, 53 courses received final approval shortly after revision. The provost’s office approved deadline extensions for the remaining courses (n = 25), which are currently under revision and due for final approval by the end of the 2015– 2016 academic year. Approximately $79,000 was provided to faculty and departments for course revisions. Furthermore, the university is now replicating the process for its graduatelevel DL courses. Moving forward, every DL course is expected to be reviewed for QA standards compliance every 3 to 5 years.

Prior to 2012, USC faculty members developed and taught courses without the aid of an instructional designer, a standardized design format, or a formal review process. However, providing professional assistance and quality assurance guidelines for faculty members to follow enables students to have improved and consistent experiences across courses. Therefore, USC faculty members now meet with an instructional designer to begin the DL course review process through an initial consultation, where the instructional designer explains the QA Standards and provides faculty with an editable online template “sandbox” course within the university’s learning management system (LMS). Although faculty members are then given full control of the content that they place within the LMS, they are checked for quality components (e.g., standard navigation, measureable learning and module objectives, rubrics to all graded assignments, ADA compliance on all videos, documents, etc.). All DL faculty members at the university are asked to adopt these guidelines as they build or redesign DL courses.

The purpose of this study was to determine faculty viewpoints on the adoption and implementation of university required quality assurance standards and look at how well the process met the goals of securing faculty compliance, buy-in, and satisfaction. Other areas investigated were: (1) perceived importance of having courses reviewed by the university; (2) adequacy of instructional design training and technical support; (3) sufficiency of the LMS or Web 2.0 tools available to enhance the course; and (4) whether student response to the course improved after review and revisions.

Effective DL course development requires faculty to embrace both changes in pedagogical approach and delivery technologies, and to also shift away from the notion of working in isolation. In particular, the importance of faculty teaming up with instructional design professionals has long been deemed important in DL course development (Ellis & Phelps, 2000; Kember & Mezger, 1990). With a steady growth rate projected at 13% annually, the primary function of instructional designers is to assist in the technology-based redesign of courses and assist faculty in learning about, becoming proficient in, and applying instructional technology (Bureau of Labor Statistics, 2012). This team-based model of course development enables members to share know-how and experiences, fostering faculty development and promoting innovative design developments (Ellis & Phelps, 2000).

An LMS is a self-contained webpage with embedded instructional tools that permit faculty to organize academic content and engage students in their learning (Laster, 2005). Those electronic forums often include a discussion board, grade book, announcement board, assessment tools, multimedia resources, and messaging systems. The use of LMS in higher education is relatively new (e.g., the Blackboard Learning Management System was introduced in 1995) (Gautreau, 2011) and unless mandated to do so, faculty have adopted LMS technologies at varying levels. However, according to Georgouli, Skalkidis, and Guerreiro (2008), faculty members who use an LMS “can improve quality of the learning experience, increase the availability and accessibility of learning materials, support collaborative activities and strengthen the feeling of belonging to an academic community” (p. 229).

Initially developed under a federally funded grant, the Quality Matters (QM) program is a leader in quality assurance for online education. The QM rubric is based on nationally recognized standards of best practice, research literature, and quality assurance standards for online course design (Shattuck, 2007), and has received wide recognition for its scalable, peer-based approach toward continuous improvement in online education. Subscribing institutions may have their courses reviewed by external QM-certified faculty reviewers, or may conduct their own internal reviews based on the rubric. Despite the benefits of such review, at least one study suggests that even where some financial incentive is offered, when given the option, most faculty will choose not to have their DL courses reviewed for compliance with established quality standards (Altman, Schwegler, Bunkowski, & Gazi, 2013).

Although most often linked to new technology, an “innovation” could be a new idea, project, or practice (Rogers, 2003). When faculty view the full use of innovation as the best course of action, they are deemed to have adopted it (Rogers, 2003). Faculty will adopt an innovation more quickly if it offers advantage, compatibility, simplicity, and observable results; even when these elements are met, implementing institutional change can be difficult (Rogers, 2003).

Among other things, faculty motivation to adopt innovation is influenced by availability of resources, a system of rewards and incentives, and administrative training and support (Smylie, 1988). These may include, among other things, release time, access to innovative software programs, technical support, and professional development support (Quick & Davies, 1999). Similarly, a sense of responsibility, personal achievement, pay, and institutional mandate factor into faculty motivation to adopt a new system (Gautreau, 2011).

Often university and college faculty are resistant to curricular changes, as some view these changes as “an administrative whim or an educational fad that will pass if ignored long enough” (Lane, 2007, p. 87). However, as new generations of students enter these universities and colleges, faculty members have to adopt new curricular (pedagogical) changes in order to successfully transfer knowledge. According to Hainline, Gaines, Feather, Padilla, and Terry (2010), two key motivators that administrators can provide to faculty to help improve buy-in is to “be willing to commit funds for faculty release time” (p. 5) and “create mechanisms to give faculty from different disciplines course credit for team teaching” (p. 5). By having release time, faculty members can devote to learning more about the curricular changes (without the burden of a set number of courses to teach), along with networking with fellow faculty on determining how to best implement these curricular changes. Lane (2007) suggests creating a participatory process for curricular changes. Often those faculty members “who are not invested in the innovation are included to focus on what they may lose rather than on what students may gain” (Lane, 2007, p. 89). By allowing for a participatory process, those faculty members who are interested can be the first to adopt and then share their experiences with fellow faculty members.

Little research exists regarding faculty response to implementation of quality assurance standards to the development and revision of DL courses. This study adds to the body of knowledge regarding effective implementation of a quality assurance protocol in undergraduate online education.

The framework used throughout this study was based on the technology acceptance model developed by Davis, Bagozzi, and Warshaw (1989). Technology acceptance model is a framework “used to provide an explanation of the determinants of computer acceptance that is general, capable of explaining user behavior [i.e., use] across a broad range of end-user computing technologies and user population” (p. 985). This model focuses on a user’s perceived ease of use and usefulness, which, according to the model, determines an individual’s intention to adopt an innovation. Overall, the technology acceptance model suggests that if the user’s perceived ease of use and usefulness are high, the user will adopt the innovation. The innovation in this study was the set of QA standards adopted by the university and the review process itself, which requires faculty to go about developing their courses in a different way.

In this study, both perceived ease of use and usefulness were evaluated through a series of focus group interviews. The researchers wanted to determine whether faculty members found the QA standards beneficial and easy to use, and to what degree they found the review process itself user-friendly and beneficial. The results were intended for making programmatic improvements and to direct further study.

Participants in this study consisted of faculty members, both full-time and adjunct, at various campuses within the USC system. These faculty members taught undergraduate courses in a DL format and worked with instructional designers within CTE to develop or modify one or more courses based on the university’s QA standards for DL course development.

Faculty members were selected in equal proportions from three groups: (a) those who developed a new online course; (b) those whose existing online course was required to go through a quality review process; and (c) those whose course was offered through Palmetto College, the university’s online college for degree completion. Faculty members in Group A applied and received a grant up to $8,000 to convert an existing face-to-face course to a 100% online format. Faculty members in Group B were required to update their existing online course to meet the university’s quality assurance metric and received a $1,500 grant after the course passed quality review. Faculty members in Group C either revised an existing online course or built a new online course for Palmetto College and received $2,000 and $5,000 respectively.

After obtaining Institutional Review Board approval, the researchers were provided a list of faculty who had participated in CTE’s instructional design consultation services and had an online course reviewed and approved as compliant with university QA standards. Upon receipt of the list of faculty, the researchers sorted the list by the three types of groups. Then, up to 10 faculty per group were randomly selected to participate. Subjects were contacted initially through an e-mail requesting participation. Nonresponsive invitees were contacted through telephone recruitment calls, using the same invitation. To obtain the sample, the researchers utilized a list of qualifying faculty members (n = 33) in each category provided by CTE. Using a random number generator, ten participants were selected from each category and sent an invitation to complete the survey. Participants who completed the survey were then invited to participate in the focus group.

Two faculty from each group completed the survey and attended the focus group, for a total of six participants. Those who agreed to participate had taught and revised multiple courses. Table 1 below highlights the demographic information of the sample.

The study consisted of a mixed-method design, primarily administered through an online questionnaire survey and three focus group sessions. Subjects were asked to complete a short online preinterview questionnaire and to participate in an hour-long anonymous group interview conducted through AdobeConnect online conferencing system. The participants utilized the chat feature of AdobeConnect to type responses to 16 openended questions. The researchers recorded the discussion/interviews to ensure that accurate details were captured. The open-ended data were analyzed for key themes while the online questionnaire items were analyzed for descriptive statistics for subject demographics.

Focus group participants were asked a series of questions regarding their perception of the QM-based internal QA standards used to guide their course design and review. All participants agreed that it is very important to an educational program to have online courses reviewed for quality. Only one participant indicated that they had prior knowledge of course quality assurance rubrics before working with an instructional designer, and that participant had only heard about it in passing.

Three participants responded that using the QA standards enhanced their effectiveness on the job, while one was not sure. When asked if the QA standards were easy to use, three respondents indicated positively, one of whom regarded the standards as “pretty straightforward.” Another indicated that once the instructional designer clarified the standards, they “made sense.” When asked if using the standards was important and/or relevant, four of the six responded positively.

Four subjects agreed that implementing the QA standards garnered results that they could see, and one said they helped with “flow.” Another indicated that the course was now more navigable and accessible, and a third noted that the standards provided for better organization, making it easier to facilitate in a compressed time frame, even though it took more time to develop a course under the QA standards than before they were implemented. The fourth positive response explained that using the QA standards helped students perform better in the course.

Participants were asked if they were interested in or concerned about relating the QA standards to what other faculty are doing. Three of the six responded either yes or that they had thought about it. Two said no, not really. The sixth said it did not apply to them. Finally, the faculty participants were asked if they were finding variations to improve upon the QA standards, and none had done so.

Faculty were asked to discuss their thoughts on the LMS course template that was provided for their new course or redesign. Specifically, they were asked whether: (a) they enjoyed using it; (b) it met their personal standards; (c) creating content around the course template hindered their course design; and (d) having a template made course development easier or more difficult. Four of the six provided favorable responses, with one stating that it was an enormous help and provided a nice foundation for building the course. Two did not enjoy using the LMS course template; one found it tedious to manage and the other felt it restricted their freedom.

Participants were asked whether they were satisfied with the level of support received from their instructional designer from the Center for Teaching Excellence and the University Technology Services (UTS) unit, which is a separate unit from the Center for Teaching Excellence. Participants indicated that they were satisfied with the level of support received throughout the course development and review process. However, two faculty members suggested that in the future, instructional designers might provide a general schedule with steps for faculty to follow during the design or revision process. One noted that, “intermittent follow-ups would help in the process to ensure there is proper development occurring.” Another suggested that when designing a course for the first time they “would have wanted a schedule with clear milestones.” As to the level of support received by UTS, four out of the six participants stated they essentially did not use UTS resources during the course development and review process. Rather, they directed technical questions to their assigned instructional designer. Those who used UTS did so only to solve students’ technical issues.

Participants were asked if they needed to adopt or learn any new technologies during the design and review process. Three indicated that they did, and mentioned Adobe Presenter, VoiceThread, and Cengage MindTap in particular. One added that they “discover[ed] so many ways of using software like Microsoft Word to make content accessible.”

Overall, faculty were not extremely satisfied with the grant compensation offered to make their courses QA compliant. Many felt that the time taken to make the course accessible under the QA standards was too great compared to the grant funds provided. Specifically, transcribing already-recorded video- and Power Point-lectures (required to meet accessibility standards) was seen by most as an onerous task. One participant voiced concern that the grant money she expected to receive as compensation was used instead by the department to pay a teaching assistant to transcribe course lectures.

Respondents were asked to compare student course satisfaction before and after implementing the QA standards. Some participants saw improved course evaluations from students, while others saw student course evaluation scoring drop. One instructor stated that despite it being completely online, his course was negatively evaluated on the line item asking whether “sufficient office hours” were provided. As to instructor satisfaction before and after implementing the QA standards, most were either satisfied or very satisfied with the results. Some added that they personally missed the face-to-face interactions available only in traditional courses.

In this study’s examination of faculty perspective on implementing quality assurance standards to DL courses, several overarching concepts were identified, including the need for incentives, setting a timeline of benchmarks, routine follow-up with faculty during the process, training on the quality assurance standards, and technical/design support throughout the process. Each of these areas were deemed important to those interviewed, and appeared to impact their perception of the process.

Incentives and compensation seemed to be at the forefront of conversation. Participants believed they should be compensated either monetarily or with course release time to redesign existing DL courses. While the quality review process outlined a small stipend for faculty, some stipends were allocated directly to departments and some participants did not personally receive this benefit. Even so, most participants did not initially recognize the amount of time it would take to design or redesign an online course. As such, several did not begin the project expeditiously, causing an increased workload toward the end of the revision period. A suggestion was offered that the designer and faculty member set milestones earlier in the design process to help faculty stay on track. Along those lines, participants appreciated their designers who periodically contacted them to check in and offer assistance.

Although a period of time had passed since the courses were reviewed, many participants seemed unclear about the specifics of the QA standards. One influencing factor may have been that at the time of course review, these standards were sometimes inaccurately referred to by some as “the QM Rubric.” This apparent confusion underscores the need for designers to use consistent terminology for the QA standards throughout the process. In addition, instructional designers should review the QA standards with faculty periodically, and reference the applicable standard during course design and review, which might help faculty better retain information about them.

Faculty participants interviewed recognized the need for additional Blackboard system training and support. They requested training sessions that would not conflict with their teaching schedules, or could be made available in a webinar format. All recognized the need for more training and support on the use of this LMS. Further, the need for administrative recognition of the importance of teaching and professional development, especially in the tenure and promotion process, was emphasized. Some participants noted that if the institution’s administration does not value teaching and professional development, faculty will not focus their efforts in those areas.

Overall, participants whose courses were either designed or redesigned using the QA standards indicated that the process was somewhat “fuzzy” for them the first time it was used for course design or redesign, but the second time the process seemed easier. Although faculty members in this study had never worked with a designer before, they recognized the importance and value of working with a professional instructional designer in the QA review process. They appreciated the individualized assistance, expertise, and effective responses received.

Although sample size of this pilot is a limitation to the generalizability of the results, the results will help generate future questions for further study. Through this pilot study, several lessons were learned that could help improve institutional quality assurance reviews, as well as instructional design consultations. There is great value in having faculty involved in the review process, garnering their support and thereby increasing the likelihood of positive change. However, when faculty feel they are mandated to modify their courses, they tend to expect additional pay or time release. Thus, faculty see required course modification as additional compensable work. Also, designers must be consistent in how they reference the set of QA standards, explain clearly what the standards are, and provide examples of how they can be achieved. Faculty participants who better understood the QA standards and review process seemed to exhibit greater satisfaction.

Instructional designers are a key element in the success of a quality assurance program implementation. Their role should not be underestimated in this process, nor the value they provide through individualized faculty coaching and assistance during course development and implementation. Institutions must invest in professional support personnel and understand that instructional designers must be highly skilled not only in online technology, but also in pedagogical and instructional design theories. Based on the results of this study USC’s model of structuring its instructional design team as professional staff within its Center for Teaching Excellence, an office which reports directly to the provost’s administration, appears to be effective in providing support to faculty. While the main focus of this study was faculty satisfaction, by looking directly at how faculty experienced the quality review and redesign process and to what degree they accept and perceive benefits (buyin), future research should build upon and delve more deeply into effective ways to ensure the quality of online course offerings, institutionwide, through internal quality assurance reviews.