Perfect Storm for the Quality Course Review at UCF Free

Can anyone easily define “quality” in online courses? Is it a nebulous term that is just a marketing ploy to attract students, or are there standards and precedents set for a common understanding? Current online courses are now considered as much of traditional higher education as brick and mortar face-to-face instruction; in fact, online learning has expanded to the point that a student from one state can get a full degree online offered from a higher education institution located in another state. Students within the Florida State University System can take online classes from multiple institutions within Florida and earn one degree. The flexibility and choice is great in some aspects, but it can make institutions nervous regarding enrollments and retention. What if students start enrolling in classes at another university because the reputation is deemed “better”—or perhaps more troubling—”easier”? Often students choose to enroll in online higher education based on the brand recognition of the institution. For online college students, the top three most important factors in choosing which school to enroll, behind tuition and fees, are the reputation of the program and school as reported most frequently by Clinefelter and Aslanian in 2017. Nevertheless, quality online courses are increasingly difficult to distinguish. Some universities with excellent brands use no quality assurance system or employ outside organizations and faculty to contribute to components of their online learning programs. If the programs and courses do not live up to an institution’s reputation, then students, because of the proliferation of options, have a greater opportunity than ever to choose a different online offering, literally, with a click of a mouse. As a mechanism for upholding the mission of the University of Central Florida (UCF) and its stellar reputation for delivering quality online and hybrid courses, UCF began a quality initiative. The aim was to verify that online courses at UCF met standards for quality, informed by best practices and nationally recognized quality course standards.

In the past, measuring quality within the educational literature was primarily achieved by comparing online course data to that of face-to-face courses. Doing so assumes the premise that face-to-face courses set the standards in which to be compared, which is not necessarily the case (Mitchell, 2010). More recently, quality in online education has been often measured in two ways: student data (satisfaction ratings and attrition), and adherence to course design standards and process (Lenert & Janes, 2017).

Online courses contain many components and often vary from each other greatly, resulting in a lack of consensus on what constitutes a “quality” online course (Thompson, 2008). Several companies have arisen to take on the massive task of creating standards that model quality. Although there are differences between the companies’ evaluations, the standards are similar overall and generally focus on best practices (Baldwin, Ching, & Hsu, 2017). There is some evidence in the literature to suggest that building a course using nationally recognized standards results in higher student perceptions (Crews, Bordonana, & Wilkinson, 2017) and overall improvement of online courses (Baldwin et al., 2017). It is important to note that the implementation of the standards is key. For instance, Bowser, Davis, Singleton, and Small (2017) implemented a five-phase collaborative review process to improve course development and delivery which involved multiple stakeholders. Professors self-assessed the quality of their course using Quality Matters standards, and peer-reviewed others’ courses. Recommendations for revisions were made at a department level. This collaborative process improved the quality of online programs within the college and has been designated as a “best practice” in the education field.

These findings suggest that reviewing a course for quality should result in a better online course to enhance both faculty and student satisfaction, as well as student learning outcomes.

It is important to note that there is not one magic factor that defines a quality experience. When measuring quality assurance strategies, “administrative leadership and support, ongoing program concerns, course development, student concerns and needs, and faculty development” should all be considered (Lee & Dziuban, 2002).

The purpose of conducting quality reviews is to assess whether a course meets a set of standards from which quality can be judged. This set of standards can be utilized by experts in online course design and teaching as a standard to judge course quality across institutions. When a course meets quality standards, not only does this justify that the rigor of online courses compare to face-to-face courses, but it also provides professional development to those (faculty members and instructional designers, perhaps) contributing to the course production process. Those participants share with each other best practices for and trends in course design and delivery, which can impact practices beyond just the online course. From participation in course review processes, a large majority of participants either had enhanced or planned to enhance their online courses, as well as face-to-face courses (MarylandOnline, Inc., 2017b). Both the awareness of quality criteria and participation in the review of quality course criteria raise the quality of not solely online, but also the face-to-face curriculum.

The University of Maryland was one of the first to create and popularize a quality course review process for reviewing online and blended courses. The small group of colleagues in the MarlyandOnline, Inc. consortium successfully launched Quality Matters with grant funds to develop a method for assessing the quality of a course across modalities or course origination through training peer reviewers and conducting course reviews nationally (MarylandOnline, Inc., 2017a). The group received a grant to fund the implementation of their goal. Quality Matters persists today as scalable quality assurance programs with quality rubrics, derived from literature reviews about online learning and best practices of course developers and instructors.

The Quality Matters quality assurance systems encompass just a few of the many quality review systems available. Many other quality systems, usually encompassing a rubric and a course review process, emerged that focus specifically on online course design quality (e.g., State University of New York, Open SUNY Center for Online Teaching Excellence’s The Open SUNY COTE Quality Review Process and Rubric, California State University’s Quality Online Learning and Teaching, and University of Florida’s Standards and Markers of Excellence Review Form); some specifically focus on blended or e-learning courses rather than fully online courses (e.g., Center for Distributed Learning Blendkit Course’s Blended Course Self-assessment/Peer-review Form, New Mexico State University’s Online Course Design Rubric); and others focus on quality delivery of instruction (e.g., Online Learning Consortium’s Quality Course Teaching and Instructional Practice Scorecard and California State University’s Rubric for Online Instruction). Although various assessments are available to review quality components, the quality assurance system at UCF focuses on quality course design in higher education.

Since its inception in 1995, with the very first online course offered a year later, UCF has invested in online learning to ensure quality in the course development process and outcomes. Early efforts to create a community of online faculty was accomplished through faculty development training (IDL6543) and the Pegasus Connections, “a compilation of resources for faculty and students which contained numerous tutorials.… Students [faculty participants] were able to use the disc to learn new skills, improve existing skills, or check their knowledge” (Truman, Futch, Thompson, & Yonekura, 1999).

Though IDL6543 has been redesigned a few times to remain current in content and design, its focus continues to engage participants on three principal tenets of quality: pedagogy, technology, and support (see Figure 1). While the Center for Distributed Learning (CDL) at UCF has many support teams to provide technical and course production assistance for faculty, the relationship between the faculty and instructional designers is the primary mechanism for developing quality courses. Other professional development opportunities are offered to faculty as well, in efforts to provide best practices for quality online course development, to include targeted topical formats such as faculty seminars in online teaching.

The issue of quality course development has historically been addressed in the training process through the course content, small groups, student and faculty panels, and the instructional design consultations. It appears to have paid off—students and faculty involved with online and blended courses at UCF have consistently reflected high satisfaction rates as found by Dziuban and Moskal (2016; see Appendix A). The researchers also found higher student success rates and lower rates of withdrawal when comparing online and blended courses to their traditional face-to-face counterparts (Dziuban & Moskal, 2017; see Appendix B).

Although this research supports the idea that most faculty are already developing courses of quality design, the relationship forged with the instructional designer during IDL6543 is not always sustained once the participant completes the program. It has primarily been the faculty member’s responsibility to reach out when needed, but instructors can teach without supervision or guidance if they so choose. The next step in promoting quality for UCF is creating a systematic method to guide optimal course design over the years through a course review process, while continuing to build relationships with faculty in a proactive and deliberate manner—a critical aspect of this initiative. Professional development programs, such as IDL6543, will embed the quality course process within the program itself, along with existing online course award programs, such as, Chuck D. Dziuban Award for Excellence in Online Teaching. This will create formal opportunities to incorporate specific processes that designate online courses as quality or high quality to effectively reach the goals of the SACS Accreditation, State University System 2025 Strategic Plan for Online Education, and the UCF Collective Impact Strategic Plan. These plans and commitments share a unifying goal—identifying quality online courses—and have resulted in CDL Strategic Initiatives with core components focused on faculty development and quality (see Figure 2).

A quality initiative began at UCF to identify and implement strategies that would contribute to enhanced online courses, the primary strategy being the establishment of a course review process. To accomplish this, a task force of instructional designers reviewed several nationally recognized quality rubrics: Quality Matters, Quality Online Learning and Teaching, California State University Chico Rubric for Online Instruction, Open SUNY COTE Quality Review, along with a review and incorporation of existing UCF course standards used for credentialing faculty to teach online. The task force developed several iterations of the Quality Online Course Review (QORC), while carefully considering the applicability of the items and the culture of faculty across departments. To verify the instrument, the QOCR was compared to the essential standards of the Quality Matters Higher Education rubric, and it was modified to address the full intent of those Quality Matters essential items. Within 6 months of the initiation of the Quality Initiative, the enhancements from the iterations and Quality Matters crosswalk enhancements confirmed the QOCR rubric as equitable to nationally recognized standards of quality (see Appendix C).

Cross-functional support was sought from CDL to standardize and scale the course review process. An instructional designer created an instrument for ease of scoring and explaining the QOCR items during a quality course review, and a developer modified an existing database for documenting the process. The task force sought the expertise of technical support staff to expand their current role to include key course development efforts for assisting faculty with revising their courses to meet quality standards, as the intent is for all courses reviewed to ultimately meet QOCR’s quality designation. Simultaneously, task force members collaborated with graphic designers and developers to develop a badging system to designate courses as having met quality standards, which would enable the placement of a time- and date-stamped logo in online courses representing quality. With a quality process standardized, support for course development in place, and faculty champions, the QOCR process will continually improve and facilitate course improvement across the University as it grows in popularity, as an established scalable and sustainable quality course design assurance system at UCF.

Communication with instructors is a critical component to the successful rollout of the course review process (Table 1). From the beginning, it was stressed that instructors would have the chance to review and have input into the review instrument. Once the QOCR was initially drafted, it was presented to a group of instructors who attended a campus conference session about quality in online courses. In the session, they were provided details about the state’s strategic plan and given some time to ask the questions that immediately came to mind for them (for instance, “Is this mandatory?,” “Will my SPIs [Student Perception of Instruction evaluation] influence this review process?”, “Will students start taking online classes at another school because they are marked as Quality?”). Once those were acknowledged and addressed, the draft of QORC items was distributed for the instructors to review. It was interesting to note that the general consensus was, “Oh, this isn’t so bad.” They were also given the chance to sign up to participate in the course review process, as beta testers of sorts. The Quality Initiative manager continued to garner support and feedback from faculty for the QOCR process by presenting at high profile University events as well as the CDL Faculty Advisory Board. Members of CDL executive leadership are also sharing the message by visiting deans and chairs to garner higher level department support.

As is the case with any professional development, providing incentives is important to getting instructors involved. Currently, we do not offer formal incentives like stipends or course releases. However, we do offer items that can be valued, such as a congratulatory letter that is suitable for dossiers, attractive badges which can be placed within the online course for students to see, and a mention that the course will be recognized as “Quality” within the state’s online course catalog. Anecdotally, we have heard of people interested in participating because they seek the “competitive edge” in bolstering student enrollments.

While communication with instructors is crucial, implementation of the review process will not be successful without “buy-in” from the instructional designers and other support staff (Table 2). At UCF, we have nearly 20 instructional designers, and 60 additional support staff who could potentially aid in the review process (course development, graphics, etc.).

Questions the task force first encountered from support staff were not unexpected. Instructional designers were concerned with the time commitment it would take to participate in the review process, when their plates were already full. Concerns about expectations also arose; how many reviews were expected to be completed each semester? How many instructors with which to engage? Are there consequences if the goals are not reached?

To address some of the initial concerns, overall framing of the process was necessary. It was emphasized that we have been engaging in course reviews to some degree through our interactions with faculty already. Formalizing a review process makes it more systematic, and gives us a common ground/language/framework in which to work. It also gives us additional points for entry in building relationships with faculty, which is a core function of the instructional designer role at UCF. In addition, a series of team goals was set rather than an individual goal, with the intention of promoting cooperation and camaraderie. In addition, each ID was paired with one member of the task force, to gain further support. For the first semester implemented, the team goal was set to average one review per instructional designer, with a “stretch goal” of two reviews. Once reaching the stretch goal, an incentive will be offered (celebratory social event), funded by leadership.

Finally, training was provided in a step-by-step manner, putting the ID in the center of the process; “I’m an ID and about to embark on the course review process. What do I do first?” Several training sessions were provided in order to ensure all IDs could attend. The Quality Initiative manager also met with other coordinating teams to ensure that the purpose of the course reviews was understood.

There is no doubt that quality in an online course is the primary concern of all the players involved from students to administration. As the implementation of the quality initiative moves forward, the challenges of policy formation and revision will play a major role in the improvement of the quality course designations at UCF. The key to the implementation is malleability and engagement on all levels as the adoption process plays out. If quality can be defined in quantitative deliverables with the correct branding, the attraction will be there for students as well as faculty to reach that badging standard. Communicating and implementing a quality course review process has the propensity to strengthen the quality of learning across the university, in both online and brick and mortar courses. Moreover, the brand and reputation of a university relies on the quality of learning, assured by a quality course review system.

This article provides the basic blueprint of implementing a course review process. We share the recommendations and lesson learned so far with others who are interested in similar opportunities. Future goals include introducing a course review process that would build on the current QOCR by focusing on “high quality” standards. Other important work includes ensuring alignment of the course review processes with local and state awards and professional development. Going beyond singular course review, we would also like to explore approaching an entire online program course review, in which all of the online courses within the program would be involved.

The course provides a clear starting point for students to begin accessing vital course components, such as syllabus, course schedule, course content, and assignments.

• The syllabus includes the following provost-required course information:

  • Course title and number;

  • Credit hours;

  • Course modality (W/M);

  • Name(s) of instructor(s);

  • Methods of contact (e.g., email address, phone number);

  • Office location;

  • Office hours (face-to-face or online);

  • Course description and purpose;

  • Course objectives and/or goals suited to level of course;

  • Course objectives and/or goals are measurable and clearly stated from learners’ perspective;

  • Required and optional texts/course materials;

  • Grading policies (point/percentage breakdown of assignments, grading scale); and

  • Exam makeup policy.

• Information about academic integrity/ honesty (UCF Golden Rule), campus policies, and FERPA are provided.

• Up-to-date information for students with disabilities to connect with UCF’s Office of Student Accessibility Services (formerly Student Disability Services) is provided.

• General technical support information is provided for students and how to obtain (e.g., Webcourses@UCF Support contact information, Service Desk contact information).

• Online etiquette (“netiquette”) expectations for course communication are clearly stated (e.g., discussion boards, email, chat, web conference).

• Expectations for instructor response time and feedback are clearly stated (e.g., questions, email, assignment feedback).

• Students are offered the opportunity to meet the instructor (e.g., introduction video, written instructor bio).

• The course has an explicit pace (e.g., a schedule) to which the students are introduced.

• The course has an explicit structure (e.g., organized in modules, units, and/ or topics; tools not pertinent to the course are hidden in the menu) with a logical efficient navigation system throughout.

• The course offers a variety of instructional materials and media (e.g., external readings, assignments, discussions, videos) aligned with learning objectives and/or goals.

• Content is displayed in ways that support learning (e.g., chunking, Pages as opposed to Word docs and PDFs, etc.).

• The course offers opportunities for students to engage with the content, stating connection to learning activities or assessments, to enhance learning.

• Technical support information (e.g. tutorials, instructions) for using technology tools are provided.

• Module objectives and/or goals are measurable, clearly stated from the learner’s perspective, and aligned with course objectives and/or goals.

• Module objectives and/or goals are aligned with learning activities and assessments.

• Grading criteria for each learning activity is described (e.g. rubrics).

• Multiple methods and opportunities for students to demonstrate learning are offered.

• Technology tools support learning objectives and/or goals.

• Technology tools support a variety of interactions (e.g., student-to-student, student-to-content, student-to-instructor).

• The course offers opportunities for students to interact with other students to enhance learning (e.g., discussions, group work).

• The course offers opportunities for students to interact with the instructor to enhance learning (related to moderation/facilitation aspect).