Systematic Review of Research on Graduate Online Education from 2000 to 2020

Students in higher education enroll in online learning for several reasons. The main reasons graduate students prefer online education are that they work full time, effectively manage their time, and have accessibility and flexibility (Ilgaz & Gulbahar, 2017). Although many of the reasons identified for undergraduate and graduate students were the same, Ilgaz and Gulbahar (2017) noted that lifelong learning was a theme associated only with graduate students. In addition, being married, having a family, and living away from the university but still being able to participate were reasons for graduate students’ preference for online learning. Graduate online learners might thus have different needs and reasons for participating in graduate online education. The design, facilitation, and assessment of online educational experiences and support structures for graduate online learners who are working professionals differ from online education for traditional learners and have been highlighted in prior research on graduate online education (Pollard & Kumar, 2021).

Although many reviews have been published on various aspects of online education and online learning (Martin et al., 2020; Kew & Tasir, 2021; Wong et al., 2019), few reviews have been published that are specific to graduate online education. Gazza and Hunker (2014) reviewed 23 articles to identify the strategies to support the retention of graduate nursing students. The retention strategies identified included social presence, program/ course quality, and individual student characteristics. Social presence included strategies such as education around relationship-building, creating an online lounge where students can socialize, and welcoming students before the course begins. Program/course quality recommendations included establishing clear admission criteria, publishing the technical skills needed to succeed, and soliciting student feedback. Individual student characteristics comprised recommendations to identify at-risk students, working with students who withdraw to plan their return, and a holistic approach to admission criteria.

A systematic review of 24 articles examining effective pedagogical practices in graduate online education by Pardino et al. (2018) revealed that the best pedagogical practices for instructors were to have good interaction with their learners, including good communication and relations. In addition, they also found that it was important for instructors to adapt their pedagogy to the students to have better educational outcomes.

Srinivasa et al. (2020) conducted a systematic narrative review on the use of online videos for teaching procedural skills to postgraduate medical students. Twenty articles met the inclusion criteria and were related to online videos, postgraduate students in the medical field, and procedural skills. Using thematic analysis, they identified six themes: online videos improve procedural skills; applications of online videos; platforms used to publish and view online videos; learner factors; characteristics of online videos; and data protection.

Pollard and Kumar (2021) conducted a literature review of online graduate student mentoring between 2003 and 2019. They selected 28 empirical studies on faculty-student one-on-one mentoring and the mentor-mentee relationship between faculty and graduate student. Four literature reviews were also included. Articles were coded to describe general details, positive aspects of online mentoring, challenges, best practices, and factors that influence online mentoring. They found that online mentoring was as effective as face-to-face mentoring and positively impacted students and faculty. The biggest challenge is clear communication between mentor and mentee and the anxiety generated by this online relationship due to the lack of social presence. Best practices to support students included providing structure, fostering communities, and providing technological support. Factors that influence online mentoring include trust, similarities in values, empathy, mentor presence, workload, and prior experiences. Table 1 summarizes the reviews focusing on graduate online education.

The above systematic reviews focused on specific aspects of online graduate education, but no review broadly reviewed graduate online education research. This review attempts to do so because the needs of graduate online learners are different from undergraduate online students (Ilgaz & Gulbahar, 2017). A review focusing on this population of learners would be beneficial to understand graduate learners. In this review, we address the following research questions.

1. What are the publication trends and outlets of research on graduate online education (years of publication, journals that publish research on graduate online education)?

2. What are the research methods (country, participant level, discipline) used in published research on graduate online education?

3. What are the research themes in the published research on graduate online education?

The five-step systematic review process described in the U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse Procedures and Standards Handbook, Version 4.1 (2020) was used in this systematic review. This process includes (a) developing the review protocol, (b) identifying relevant literature, (c) screening studies, (d) reviewing articles, and (e) reporting findings. Using this process assisted us in being consistent, objective, and transparent during the review and ensured coverage of the literature as recommended.

A search for research on graduate online education was conducted in December 2020 using Academic Search Complete, APA PsycArticles, APA PsycInfo, CINAHL Plus with Full Text, Communication & Mass Media Complete, Education Research Complete, Educational Administration Abstracts, ERIC, Library, Information Science & Technology Abstracts with Full Text databases indexed in EBSCOHOST. Two decades of peer-reviewed publications from 2000 to 2020 were searched using the keywords “graduate” and “online.”

A screening criterion for inclusion and exclusion was developed (Table 2).

The PRISMA flow chart (Figure 1) was used to guide the selection of the articles from identification, screening, selection, and inclu-sion. Of the initial 990 articles, finally, 271 articles were included in this review.

A review protocol was created in Google Sheets to assist with the review and coding. The protocol included study ID, publication year, author, research theme, journal title, subject, student level, country, research method, and notes (see Table 3). The first and second authors coded the articles for this study, meeting biweekly to discuss any queries to agree on.

Martin et al. (2020) identified 12 themes in their systematic review on online learning across all levels of education. Their themes were used to code the articles in this systematic review on graduate online education., along with four additional themes-needs and challenges; modality comparison; general perception; coaching, mentoring, and advising. Further, assessment and evaluation from their review were combined, and instructor was updated to include instructor and library professionals. These changes resulted in 15 themes (Table 4) used to code articles in this systematic review.

A timeline was generated to showcase the years when articles on graduate online education were published. Frequency tables were generated for publication outlets, research context, participants, discipline, methods, and research themes. A crosstab analysis was conducted to examine the research themes by research methods and decade.

The results section includes graduate online education findings, including publication trends and outlets, research methodology components, and research themes.

Publication trends are depicted by a line graph, as shown in Figure 2. There has been a steady increase in the number of publications until 2013, after which another increase followed a decline. The highest number of publications on graduate online education were published in 2020.

Journals that published five or more articles in this review are included in Table 5.Quarterly Review of Distance Education published the highest number of articles (13 articles), followed by International Review of Research in Open and Distributed Learning (12 articles).

This section includes research methodology components such as research context, participants, discipline, and methods.

The research context of the articles was coded based on country (Table 6). More than half of the studies (59%) were conducted in the United States, 7% in Canada, 7% in other countries, and about 27% did not report the country where the study was conducted.

Participants in the research were coded based on the program level, certificate, master’s, or doctoral (Table 7). Some studiesincluded both master’s and doctoral students, and some included master’s and certificate students. Master’s students were the focus of 32% of the studies, followed by master’s and doctoral in 9.2% of the studies. About half of the studies (42.1%) did not mention the specific level of students. Also, about 3.7% of the studies examined graduate faculty, while another 3.7% had multiple participants.

Discipline was open-coded and then categorized based on commonality. About half of the studies were in education (50.5%), followed by health disciplines (15.1%). About 10% of the studies did not report a discipline. Table 8 describes the disciplines in the reviewed research.

Research design was coded as quantitative, qualitative, or mixed method. There were about an equal number of quantitative (39.1%) and qualitative studies (36.9%) and fewer mixed methods studies (24%). Tbl9 includes the different research methods used in the studies in this systematic review.

Fifteen research themes were identified within three areas, learner; course, program, and instructor; and organization (Table 10). Online engagement was the most studied research theme (24%) in graduate online education, followed by learner characteristics (13.7%), both of which pertain to the learner. Technologies were the most studied in the course, program, and instructor (8.9%). Very few studies (5.2%) focused on the organizational level.

The following section provides details and examples of studies within each research theme.

Online Engagement. Online engagement in graduate online education was the most studicd theme (24%). Some of the subthemes included learner community (Rogo & Portillo, 2014; Thoms et al., 2008; Walsh & Pollard, 2020), sense of community (Cornell et al., 2019; Gigliotti, 2016; Lewis et al., 2015; Trespalacios & Uribe-Florez, 2020), connectedness (Jamison & Bolliger, 2020; Joyner et al., 2014), academic socialization (Dong et al., 2019), program community (Shepherd & Bolliger, 2019), transactional distance (Alotibi & Miller, 2019), interactions (Selvaggi, 2016; Thompson et al., 2019), social networking (Boster & McCarthy, 2018; Cowan & Menchaca, 2014), social presence (Elcicek et al., 2018; Mayne & Wu, 2011), collaborative activities (Du et al., 2012; Kumi-Yeboah et al., 2017), cohort structure (Blankenship & Gibson, 2016), and online discussions (Bures et al., 2009; Swaminathan & Mulvihill, 2013).

Learner Characteristics. Graduate online learner characteristics were examined in 13.7% of studies. Some of the subthemes included struggling learners (Lehan & Babcock, 2020), first-time online students (Tseng et al., 2020), prior experience and self-efficacy (Kreth et al., 2019), graduate student attributes (Aitken et al., 2019; Rakes et al., 2013; Rowe, 2016), self-determination (Bissessar et al., 2019), learning preference and experiences (Swaggerty & Broemmel, 2017), help-seeking experiences (Koc & Liu, 2016), learner anxiety (DeVaney, 2016), motivation and self-regulation (Artino & Stephens, 2009; Rakes & Dunn, 2010), multitasking, distractions, and boundary management (Winter et al., 2010), and communication and learning style (Fearing & Riley, 2005; Rovai, 2003).

Course Technologies. The research on various course technologies in graduate online education (8.9%) focused on maker education (Hsu et al., 2018), video lectures (Reinecke & Finn, 2015), synchronous video conferencing (Foronda & Lippincott, 2014; Groen et al., 2008), social media (Allen & Nelson, 2013), mobile applications (Hsu & Ching, 2013), blogs (Freeman & Brett, 2012), wikis (Huang & Nakazawa, 2010; Park et al., 2010), virtual worlds (Burgess et al., 2010), and instant messaging (Chen Wang & Morgan, 2008).

Learning Outcomes. Various learning outcomes were studied in the graduate online education research (8.7%). Some of the learning outcomes examined were critical thinking competencies (Hermond & Tanner, 2020; Novotny, 2016), research-based learning experiences (Jacobsen et al., 2018), argumentative knowledge construction (Bayazit et al., 2018), evidence-based practice (Rojjanasrirat & Rice, 2017), online research behaviors (Cheng & Tsai, 2017), interpersonal problem-solving skills (Bekki et al., 2014), and service-learning (Carracelas-Juncal, 2013).

Martin et al. (2019) identified design, facilitation, and assessment as three essential aspects of effective online courses. These three topics design (7%), facilitation (5.2%), and assessment (5.9%), were moderately researched in this review on graduate online education.

Design. Some of the studies regarding design in graduate online education focused on learning resources (Wilhelm-Chapin & Koszalka, 2020), using case method of instruction (Casotti et al., 2013; Sanders-Smith et al., 2016), design for accelerated coursework (Trekles & Simms, 2013), design characteristics of graduate synchronous online programs (Aversa & MacCall, 2013); investigating multimodal texts (Karchmer-Klein & Shinas, 2012), and course and curriculum design (Austin et al., 2011; Theeke & Carpenter, 2020)

Assessment and Evaluation. Graduate online education research focused on various assessment and evaluation topics, including peer assessment (Wang et al., 2020), approach to preempting cheating (Sullivan, 2016), comparing supervision in tests (Ladyshewsky, 2015), badging for peer review (O’Connor & McQuigge, 2013), the use of plagiarism checker Turnitin (Kiriakidis, 2013), formative assessment through virtual community (Glassmeyer et al., 2011), and curriculum evaluation (Avery et al., 2008).

Facilitation. Several facilitation strategies were examined by researchers, such as using reflections and questioning to engage and challenge online graduate learners (Liu, 2019), various instructional approaches in online graduate online student instruction (Francis, 2014), and examining facilitating interactions among graduate students (Topper, 2005).

In addition, there were studies comparing modalities (5.2%) or focusing on general perceptions of graduate online education (5.2%).

Modality Comparison. Researchers compared online to hybrid courses and online to face-to-face courses. Hahs-Vaughn et al. (2017) compared statistical literacy as a function of online versus hybrid course delivery. McGinley et al. (2012) compared graduate student perception differences between face-to-face and online learning. Pribesh et al. (2006) compared online and face-to-face cohorts in a school media specialist program. Some researchers examined similarities in instructional quality (Topper, 2007) or the perception of efficacy (Robertson et al., 2005) in online and face-to-face courses.

General Perception. Researchers also broadly researched students’ general perceptions about graduate online courses or programs (Burns, 2013; Fedynich et al., 2015; Holzwciss et al., 2014; Perreault et al., 2008).

There were a few studies specifically focusing on the needs and challenges of graduate online learners (4.4%), graduate online instructors and librarians (3.7%), and inclusion, diversity, equity, and access (IDEA) (3.7%) topics.

Needs and Challenges. Researchers studied challenges of online graduate students, such as student isolation, especially among international students (Erichsen & Bolliger, 2011), stress levels among female online graduate students (Arric et al., 2011), and how to keep graduate management students stress free (Martinak, 2012). Researchers also studied online learner needs, such as accessing the library (Skarl & Del Bosque, 2019).

Instructor and Library Professional Topics specific to the instructors and librarians in graduate online education were also researched. Redman and Perry (2020) studied changes in graduate online instructor perceptions over the years, while Dooley et al. (2019) examined faculty professional development and instructional design to teach graduate online courses. Edwards et al. (2010) studied embedded librarians in online graduate research courses.

IDEA. Some researchers examined IDEA topics in graduate online education. Polnick et al. (2011) examined creating a classroom community to achieve gender equity in both online and face-to-face classes. Njoku and Baker (2019) studied cultural competence and health disparities in an online graduate public health course. Terras et al. (2015) studied disability accommodations in graduate online courses.

Themes such as institutional support (0.4%), leadership and management (3.7%), coaching, mentoring, and advising (3.3%) were least studied in this systematic review. Among the few studies on these themes, Jowallah (2014) examined the management of online teaching and learning spaces, while Anderson et al. (2010) studied tuition costs and faculty salaries. There is also a need for more research on graduate online student coaching, mentoring, and advising. Lehan et al. (2020) studied the relationship between academic coaching and program completion, and in another study, Lehan et al. (2018) examined the influence of academic coaching on persistence. Cross (2018) studied graduate student perceptions of online advising, while Schroeder and Terras (2015) studied the advising experiences of online adult graduate learners.Crawford et al. (2014) developed an online mentoring scale.

A crosstab analysis was conducted for research themes and methods (Tbl11). Assessment and evaluation studies and comparison studies were mainly quantitative, while studies on general perceptions of graduate online learning were qualitative. Graduate learner characteristics, online engagement, and technologies were studied using quantitative, qualitative, and mixed methods.

There has been a steady increase in the number of studies on graduate online education in the second decade (2011-2020) compared to the first decade (2001-2010). Studies on engagement, learning outcomes, design, assessment, and evaluation have increased from the first decade to the second decade,showing these research themes’ growing importance. While there was no research on coaching, mentoring, and advising in the first decade, nine studies were published on this theme in the second decade, highlighting the need for more research on this theme (See Tbl12).

There are a few limitations to this systematic review. The term graduate used as a keyword is commonly used in the United States to represent education beyond the undergraduate level, but this might not be the case in other parts of the world. For instance, the term postgraduate is used in many contexts. Using this term could have been why the research found was predominantly conducted in the United States. In addition, using the keywords "Masters" and "Doctoral" might have unearthed additional studies on graduate online learners. Second, the search for the keywords was a title-level search. If authors had studied graduate online learners but did not call it out in the title, the study might not have shown up in our search. Finally, the research themes are not mutually exclusive. Some studies had more than one focus area, but the fittest theme was selected to be the research theme. Nevertheless, the body of research provided insight into themes related to graduate online education that have been studied and areas for future research.

There has been a steady increase in the research on graduate online education. About half of the research was examined in the education discipline, followed by the health professions, highlighting the need for more studies in other disciplines. Given the signature pedagogies (Shulman, 2005) in various disciplines essential to the acquisition of knowledge and preparation for professions, further research in other disciplines can highlight both the various types of designs, facilitation strategies, and assessments that are valuable to learners but also learner needs and characteristics in those disciplines. Various levels of graduate online learners were represented in the research, including graduate certificates, master’s, and doctoral online students. Likewise, there was a good balance of studies using quantitative, qualitative, and mixed methods to research graduate online education.

This systematic review highlighted the research themes that were most, moderately, or least studied. Online engagement at 24.62% and learner characteristics at 13.7% were the most common themes, which is similar to the Martin et al. (2020) systematic review. They examined research on online learning across all learner levels and found online engagement and learner characteristics as the top two themes. Of the studies identified in this literature review, 59.4% were related to online teaching and assessment implementation and their outcomes-the themes of online engagement, technologies, design, assessment and evaluation, facilitation, and learning outcomes. Irrespective of the learner level, these themes are essential areas of research.

Modality comparison and general perception studies continue to be conducted even in graduate online education, comprising almost 10% of the research. The needs and challenges of graduate online learners; coaching, mentoring, and advising; and IDEA are areas of graduate online education where further research is needed. Although the design, delivery, and assessment of online learning experiences are essential for student success, understanding student needs and how they can best be coached and mentored is extremely important for this population that is different from undergraduates. Additionally, given that one of the advantages of online education is that it can reach and benefit all types of learners, further research on IDEA can ensure that educators and instructional designers better understand how to design for all learners to succeed and how specific groups of learners can best be supported in the online environment.

The findings from this systematic review have implications for graduate online program directors, graduate online program faculty, graduate online students, and higher education administrators who support graduate online education. Specifically, research findings on graduate online learner characteristics will support the design of online courses keeping learner characteristics such as prior knowledge, self-efficacy, learner preference, learner anxiety in mind. The various findings on online engagement emphasizing the importance of interaction, community, presence, collaboration, and socialization inform the design and facilitation of graduate online courses and programs. Findings from various themes such as the needs and challenges of graduate online learners benefit instructors, program directors, and administrators in offering online programs for online graduate learners. Similarly, the findings at the course level inform the design of graduate online courses.

Despite the various characteristics of online graduate learners identified in the literature, learners are diverse. More research is needed on online graduate learners from different countries, contexts, cultures, and those with varying preferences, abilities, and motivations. This review also highlights the need for future studies to examine organizational topics such as IDEA, leadership and management, and institutional support specific for graduate online education.