Assessing Online Faculty: More Than Student Surveys and Design Rubrics

Online education is proving to be highly resilient. Even in the face of the highest declines in total higher education student enrollment since the 1950s (U. S. Census Bureau, 2013), the latest surveys of online education continue to show a rise in the number of students taking online courses (Allen & Seaman, 2013). Although the number of online courses is growing, not all online courses are of equal quality. We have experienced instances of successful courses where a highly engaged and involved instructor has mitigated the effects of a poorly designed course. Conversely, we have also witnessed well-designed courses that have been taught poorly and were ultimately unsuccessful. Clearly, the quality of the individual instructor is crucial to the quality of the course (Piña, Harris, & Ashbaugh, 2013).

At professional conferences, we have heard the concerns of those who oversee distance education programs at colleges and universities that the metrics used for evaluating online instructors, often limited to end-of-course student surveys, do not provide academic deans and chairs with adequate information to make informed decisions about their online faculty, particularly adjunct faculty (Piña & Bohn,2013). In many cases, the methods that have been used to evaluate face-to-face instructors are less effective when used to evaluate online instructors (Mandernach, Donnelli, Dailey, & Schulte, 2005). Research has shown that colleges and universities tend to not do a very good job at recognizing and rewarding exceptional online teaching, particularly with professional incentives such as promotion, tenure and institutional recognition programs (Piña, 2008). A contributing factor to this situation may be the inability of institutions to adequately measure quality online teaching.

As online programs grow, the paradigm of the faculty member as developer and sole teacher of an online course is changing. Many adjunct instructors teach online courses developed by other faculty members, often in partnership with instructional designers (Cheski & Muller, 2010). At institutions with many sections of the same course, a single “master” course is sometimes used to house the course’s instructional content and assessments, which are copied into the various sections and taught by different instructors (Borgemenke, Holt, & Fish, 2013). Colleges and universities are also relying more frequently upon department- or institutionwide templates for their online courses, providing a common look, feel and navigation for students (Ley & Gannon-Cook, 2014).

Standards for online course quality have been promoted by accrediting bodies (Commission on Colleges, 2012), statewide agencies (Illinois Online Network, 2012), teachers’ unions (American Federation of Teachers, 2000) and independent education organizations (American Council on Education, 2003; Council on Higher Education Accreditation, 2002; Phipps & Merisotis, 2000). Rubrics to evaluate the quality of online courses have been formulated by a diverse group of organizations, such as Quality Matters (Maryland Online, 2008), iNacol (2012), California State University Chico (2012), Towson University (Ashcraft, McMahon, Lesh, & Tabrizi, 2008), Blackboard, Inc. (2014), and the United States Distance Learning Association (2012).

While the various online quality rubrics can provide useful direction for building online courses and assessing the quality of course design, they provide little guidance for teaching online courses and assessing the quality of online instructors. Quality online education, as measured by the most popular rubrics, tends to be focused upon the instructional design of the course, such as with Quality Matters (Maryland Online, 2008), or upon the resources and services offered by the institution, such as with the Sloan-C Quality Scorecard (Shelton, 2010; 2011).

A weakness of the existing “online teaching rubrics” is that they have been created with the assumption that the instructor who is teaching the course is the same person who developed the course and is, therefore, responsible for the course design (Innovations in Distance Education, 1998). Since an ever-increasing number of institutions are using “master” courses developed by faculty or other subject matter experts with instructional designers, many instructors are teaching online courses that they did not develop themselves. In these cases, it would be unfair to evaluate an instructor based on the how well (or how poorly) the course was designed.

This problem is exemplified by Sonoma State University’s Quality Online Learning & Teaching Rubric (Sonoma State University, 2012). The rubric contains 53 evaluation criteria; however, 49 of these criteria relate directly to the course design, while only four relate to actions undertaken by the course instructor. Park University’s Online Instructor Evaluation System recognizes the difference between course content and online pedagogy and integrates formative and summative evaluation with mentoring of new online faculty by experienced online instructors (Mandernach et al., 2005); however, it requires multiple faculty with 50% release time to mentor the new instructors throughout the course—a resource that is not available at many institutions.

So, how do we assess the quality of instructors teaching courses that they did not design? We must look at the actions performed by the instructors within the course. In face-to-face courses, evaluation of an instructor’s teaching quality is often done via a peer or supervisor observation of a live class session. This class observation method results in a “snapshot” view of what occurs during a small portion of the course (Marshall, 2012). In contrast, the online course’s learning management system preserves information about student and instructor activity throughout the course, including logged in time, course announcements, course discussions, feedback, and grades. An online class observation would provide a much more complete picture of what the instructor and students have done in the class. However, deans, program chairs, and others without significant online teaching and development experience might not know what to look for when observing an online course. It would be extremely useful to have an evaluation instrument for an online “class observation” of the instructor’s actions and teaching— with indicators of inline teaching quality that are separate and distinct from those features that are part of the course’s instructional design (Piña & Bohn, 2013).

In this study, distance learning administrators and faculty were surveyed to determine measures that are currently being used to evaluate the quality of online faculty at higher education institutions. Additionally, these professionals, along with students enrolled in online courses, were asked to validate nine observable indicators of online instructor quality.

The sample consisted of 254 total respondents, including 140 online learning professionals (evenly split between faculty teaching online courses and administrators with responsibility for online education) and 114 students enrolled in online courses.

A comprehensive review of distance education literature yielded many different sources for best practices and measures for online faculty evaluation (e.g. Mandernach et al., 2005; Tobin, 2004; Sonoma State University, 2012). In each case, the evaluation measures included items outside the control of the faculty member teaching the course, such as the instructional design of the course and policies for faculty staffing and support. There was no instrument identified that was designed to evaluate faculty apart from the course design.

A survey instrument was developed to include two items on methods that institutions use currently to assess online instructor quality, including an “other” category, where respondents could provide additional information or examples. The instrument also included nine suggested indicators of online instructor quality gathered from the literature review and from distance learning administrators and faculty at professional conferences. Each indicator was accompanied by a 4-point Likert-type scale. Respondents chose the level of importance of each item as an indicator of online instructor quality. Responses to the instrument were coded for analysis as follows: critical = 3, important but not critical = 2, minimally important = 1, and not important = 0. Respondents also selected minimum standards of instructor activity for six of the indicators.

preliminary paper-based draft of the survey instrument was pilot-tested during the 2013 Distance Learning Administration Conference with 23 professionals who were representative of the target audience. The survey items were determined by these professionals to be appropriate for the target audience, with only minor modifications recommended. The survey was distributed online via SurveyMonkey to attendees of the Distance Learning Administration Conference and to members of the Association for Educational Communications and Technology. A shorter form of the survey, excluding the items on methods that institutions use to assess online instructor quality, was distributed to students enrolled in online courses at a medium-sized private university in the Southern United States. Cronbach’s alpha for internal consistency was 69.3. Anonymity of the respondents was assured by configuring SurveyMonkey to provide each respondent with a unique identifier that would not allow the researchers to trace respondents’ identities (SurveyMonkey, 2014).

Mean scores and standard deviations were calculated for each of the indicators of online instructional quality. Scores fell between 3.0 (critical) and 0.0 (not important) for the rated importance of each indicator of faculty quality. Analysis of variance was used to test for statistically significant differences and alpha level for significance was set at P< .05. Frequencies and percentages were calculated for methods used by institutions to assess online instructor quality and for minimum standards for instructor activity.

Table 1 contains the responses to the question “Which measures does your institution use to assess the quality of online instructors?” Respondents were allowed to select multiple measures, if applicable. Student surveys were, by a significant amount, the most widely used measure for evaluating instructors. Supervisor evaluations were used by slightly more than one third of respondents’ institutions, followed by online class observations, metrics from the learning management system, and peer evaluations. Just below eight percent of institutions used no assessment measures at all and less than three percent used other measures, which included enrollment in professional development programs and self-evaluations. The fact that there were so few items in the “other” category indicates that the survey captured the domain of measures used at these institutions. Other comments offered by respondents indicated that metrics from the learning management system included reports of instructor login and discussion forum activity.

Table 2 below contains the responses to the item, “Do you use a rubric to measure online instructor quality?” Responses were almost evenly divided among Quality Matters, In-House Developed, and Do Not Use a Rubric. Twenty-two respondents selected more than one response. Comments for this survey item indicated that this reflected the use of multiple rubrics across different departments. The rubrics identified by the respondents (Quality Matters, CSU Chico ROI, USDLA, and iNacol) were all designed primarily for evaluating the instructional design of online courses, rather than the actions and contributions of the online instructor. The Sloan-C Quality Scorecard was designed to evaluate the administration of distance education programs, rather than the quality of online faculty (Piña & Bohn, 2012; Shelton, 2010). Comments offered by 13 of the respondents indicated that their in-house rubrics were based upon Quality Matters, while one respondent indicated that the in-house rubric was based upon a combination of standards from by the Association for Educational Communications and Technology and the International Society for Technology in Education. The four “other” rubrics listed were IDEA Education (2), QOCI-Quality Online Course Initiative (Illinois Online Network, 2012) and the Best Practices for Electronically Offered Degree and Certificate Programs (WICHE Cooperative for Educational Telecommunications, 2000).

Table 3 below reports the results of respondents’ ratings of importance for the indicators of online instructor quality. ANOVA revealed no significant differences between online faculty and distance learning administrators for any of the nine indicators of online teaching quality, so instructors and administrators were grouped together (n = 140) and were compared with students (n = 114). Administrators and faculty considered responding to student inquiries, providing feedback for assignments, responding in a timely manner, login frequency, and moderating discussion forums to be the most important indicators of online instructor quality. Students judged feedback on assignments, responding to student inquiries, responding in a timely manner, posting class announcements, and login frequency as the most critical indicators. ANOVA revealed significant differences between administrators/ faculty and students for five of the nine indicators. Administrators/faculty rated responding to student inquiries, timeliness of response, login frequency, and moderation of discussions significantly higher than did students, while students rated conciseness of course announcements as significantly higher. Overall, administrators and faculty had higher ratings for faculty involvement in discussion forums, while students had higher ratings for course announcements.

Six of the indicators included four possible minimum standards for instructor activity. Each respondent could select a single minimum standard for each indicator. Table 4 reports the response frequency and percentages for administrators/faculty and for students. For frequency of instructor login, no clear picture of student preference emerged, while administrators/faculty indicated that instructors should log in to their online courses either every day or every other day. Having instructors post weekly course announcements was clearly favored by both students and online professionals, with the highest responses of any standard and with a notable minority (particularly students) favoring announcements more than once per week. Nearly half of administrators/faculty preferred to have no word limits on course announcements, while students were undecided. Both groups rated this indicator as lowest overall in importance.

Responding to student inquiries within one day garnered high percentages—especially with students, followed by responses within two days. Waiting three or four days to respond was acceptable only to a minority of respondents. Students indicated that they wanted instructors to provide more robust information about themselves than merely contact information. While providing a single paragraph brief bio was most popular with both groups, more students showed a preference toward having faculty provide a more fully descriptive biography. Administrators and faculty did not offer a consistent view regarding the involvement of instructors in discussion forums but showed a higher tolerance than students for instructors not having a requirement to post.

Overwhelmingly, the institutions in our study rely upon student surveys as the primary method for assessing their online faculty. Student evaluation of teaching is a widely researched area of education and recent metaanalyses have not resulted in clear answers regarding the validity of such assessments for formative and summative evaluation purposes (e.g., Spooren, Brockx, & Mortelmans, 2013). After outlining some of the possible sources of bias in student evaluations of faculty, Baldwin and Blattner (2003) recommended that multiple methods be used to evaluate teaching quality. Several of the institutions surveyed do use additional methods, such as supervisor or peer evaluations and online class observations. These methods, however, appear often to be based upon rubrics and standards formulated to evaluate the quality of the course’s instructional design, which assumes that instructor being assessed is also responsible for the design and development of the online course. This is often not the case and may lead to judgments being made based on criteria that are not relevant to the actual role played by instructors in their online courses.

Our desire was to identify a set of criteria that would yield objective data easily examined by supervisors and peers during an online course observation and serve as a balance to the more subjective data gathered from student surveys. This study focused upon quantitative measures of instructor actions and behaviors that could be readily observed in the online course and/or collected using the reporting tools of the learning management system:

• Has the instructor logged in at least an average of every other day?

• Has the instructor posted a biography of at least a paragraph, in addition to contact info?

• Has the instructor posted announcements at least weekly?

• Is there evidence that the instructor answers student inquiries in two days or less?

• Does the instructor participate in discussion forums where appropriate?

• Does the instructor provide feedback on assignments?

We started with what could be considered the “low-hanging fruit” of easily observable and quantifiable instructor actions. More difficult will be to determine the levels of quality within these indicators. For example, what constitutes higher versus lower quality course announcements, instructor bios, feedback on assignments, or instructor discussion forum posts? What are the indicators of an instructor who successfully facilitates online discussions? How do instructors leverage their expertise to add value to an already developed online course? These questions of quality online teaching are fruitful areas for further research.

Most learning management systems are adept at generating reports that can catch student or instructors doing the wrong thing (e.g., not logging in, not posting in discussions, inappropriate communications, taking a long time to respond to inquiries or grade assignments, not providing feedback). However, these systems need to become better at catching people doing the right thing or doing exceptional things (e.g., flagging highly responsive instructors and identifying courses with high levels of interaction).

As the number of online courses continues to increase, our efforts to adequately and accurately assess and develop those who teach these courses also needs to increase. Ultimately, the quality of our student’s educational experience will depend upon these efforts.