Evaluating The Design and Development of the Quality Graduate Supervision miniMOOC

Graduate supervision is an important and demanding component of most academic faculty members’ research, teaching, and service responsibilities. Many professors tend to develop graduate supervision knowledge and competency on the job versus through active and intentional training and mentorship. While many new and more experienced professors may welcome seminars on graduate supervision, the act of balancing demanding research, teaching, and service roles and responsibilities can leave little discretionary time for scheduled faculty development seminars or workshops. To support both new and experienced graduate supervisors in continual professional learning to enhance their graduate supervision knowledge and skills, we designed a mini massive open online course (miniMOOC) on quality graduate supervision for academic faculty members at the University of Calgary (Alharbi & Jacobsen, 2016b). The first offering of the QGS miniMOOC was labeled “mini” because the first iteration was offered as a closed pilot for research at one institution. The QGS miniMOOC is an innovative design solution that provides flexible and accessible online faculty development in a networked learning environment for graduate supervisors that was part of a PhD research project.

This article reports on the design and development phases in the production of the QGS miniMOOC. As an innovative design solution, the QGS miniMOOC was part of an overall design-based research investigation into the analysis, design, and evaluation of online faculty development for graduate supervisors. This paper builds upon the analysis and exploration phase of the miniMOOC (Alharbi & Jacobsen, 2016a) and describes the design elements that were determined to be relevant and necessary for the development of this online faculty development experience. Second, this paper describes the process and challenges experienced during the development phase of the QGS miniMOOC and the evaluation of the implementation. This article focuses on reporting the descriptive data from the participants’ activities in the QGS miniMOOC. It is expected that outcomes and impacts from this design based research project can inform the design and development of online faculty development and learning opportunities in higher education. It is also expected that findings from this research will be relevant for graduate students, academic faculty across disciplines, educational developers, and for senior leaders in higher education.

Professors across ranks and across disciplines undertake the mentorship and supervision of both master’s and doctoral students in their disciplines of study within the first few years of their career. Graduate supervision can be defined as “a complex pedagogical practice. It is a partnership between an experienced and an aspiring scholar, which shifts over the number of years it takes for the research to be done and the thesis to be written” (Kamler & Thomson, 2014, p. 1). Two of the presenting challenges in successfully supervising graduate students are, one, to support graduate students, especially doctoral students, in completing their dissertations in a reasonable time frame and, two, to maintain a productive relationship between the supervisor and the student in which the student grows knowledge and confidence and the supervisor improves his/her pedagogical approach in supervision (Walker & Thomson, 2010). Most graduate supervisors develop their practice experientially; that is, by engaging in just in time, on the job learning early in their academic careers (Alharbi & Jacobsen, 2016b). Some graduate supervisors approach graduate supervision in the ways that they themselves were supervised, while others actively take steps to supervise students in different or better ways. It is clear that the quality of graduate supervision experienced by students varies (Beaudin, Emami, Palumbo, & Tran, 2015). The relationship that develops between graduate supervisors and graduate students is one of the important factors in the success of students in their graduate studies (Adkins, 2009; Skarakis-Doyle & McIntyre, 2008). One issue in developing learning opportunities for graduate supervisors is the paucity of research literature on how to best support graduate supervisors in developing their practice. A few pioneers in higher education are paying attention to the types of support and informal learning experiences that help to prepare graduate student for research and new faculty members for graduate supervision (see The Thesis Whisperer at thesiswhisperer.com, and The Supervision Whisperers at thesupervi- sionwhisperers.wordpress.com). Research on student satisfaction with graduate supervision has been carried out (Beaudin et al., 2015; Erichsen, Bolliger, & Halupa, 2014), and advice is provided on helping graduate students with writing (Kamler & Thomson, 2014) and with teaching (Altman, Stein, & Stowell, 2015); however, there is a paucity of research on how to become a good supervisor beyond identifying some characteristics of good supervisors (Ismail, Hassan, & Masek, 2014). Dangel and Tanguay (2014) wrote, “there is conceptual and empirical literature that points to the need for quality supervision; however, there is less information on how to best support supervisors in their work” (p. 4).

On-campus workshops, seminars, and sessions focused on graduate supervision can also enhance the skills of newer and more experienced supervisors, and help them to develop competencies beyond their individual experiences of being supervised. However, it is important to provide faculty development opportunities that are accessible, flexible, and responsive to faculty members’ demanding work weeks and also to their unique needs and diverse levels of experience. The busy schedules of faculty members, volume of academic work expected, logistical issues such as the times and locations of sessions, lack of recognition or financial awards for teaching, and lack of direction from the university are some of the reasons for low participation rates by professors in faculty development opportunities available at institutions (Taylor & McQuiggan, 2008). In addition to being accessible and flexible, the design of faculty development opportunities to improve graduate supervision skills should emphasize the creation of a safe space for supervisors to discuss, critique, and debate this pedagogical relationship of supervision (Manathunga, 2010).

MOOCs offer one approach to faculty development that can address accessibility and flexibility issues and thus may encourage greater participation among diverse faculty members. Researchers have recommended the use of MOOCs as a workable solution for faculty professional development (Bond, 2013; Fini, 2009). Fini (2009) explains that most people attend MOOCs for professional development reasons. Zhenghao et al. (2015) report that 52%of respondents who were surveyed after enrolling in a MOOC in Coursera identified building their career by enhancing or improving their current skills, or to find a new job as the main reasons they took a MOOC. Studies indicate that MOOCs can provide accessible professional development opportunities for faculty and can also have benefits for those who engage in this approach to professional development (Bond, 2013; Fini, 2009). Furthermore, the use of MOOCs can offer a workable solution for faculty development opportunities, as faculty can determine the amount of time and effort they put into their learning activities and the times that better fit their schedule; MOOCs can help to blur the line between professional practice and professional development if the online experience combines communication, teaching, and mentoring of faculty members (Bond, 2013).

This research is guided by two assumptions: (1) that faculty in higher education can benefit from the use of MOOCs for faculty development, and (2) a MOOC that focuses on quality graduate supervision can be designed to support the creation of a learning community among diverse faculty members. This design-based research was guided by several research questions. This paper focuses on two of the questions: “What design elements are necessary in the development of a Quality Graduate Supervision miniMOOC?” and “What scaffolding and support is necessary to support faculty members to effectively use the Quality Graduate Supervision miniMOOC?” In the next section, the approach taken in this design-based research is described with a focus on the design and development phases.

The present research involved a design-based approach to designing, developing, and evaluating a faculty development miniMOOC for quality graduate supervision. Wang and Hannafin (2005) define design-based research as “a systematic but flexible methodology aimed to improve educational practices through iterative analysis, design, development, and implementation, based on collaboration among researchers and practitioners in real-world settings, and leading to contextually sensitive design principles and theories” (pp. 6–7). This paper focuses on the design, development, and evaluation of an innovative solution, which is a focus on the second and third of four phases in this design-based research initiative: (1) analysis of practical problems by researchers and practitioners; (2) development of solutions; (3) evaluation and testing of solutions in practice; and (4) reflection to produce “design principles” (McKenney & Reeves, 2012; Reeves, 2006). In this project, the innovative solution for flexible and accessible faculty development was developed as a six-module, QGS miniMOOC, which was offered first as a closed pilot at the University of Calgary. This solution was iteratively designed, evaluated, and tested in practice. As mentioned in the introduction, this innovation was called a miniMOOC because this iteration was offered as a closed pilot for the initial evaluation phase of the research. According to McKenney and Reeves (2012), pilot studies help researchers to “begin to get a sense of how the intervention will perform in various contexts and what kind of real world realities need to be addressed for the design to have a chance of success under representative conditions” (p. 6). The longterm goal of this design based research is to build upon this closed pilot and to implement and evaluate multiple iterations from partly open to fully open offerings of the QGS MOOC over time and to eventually offer it globally to interested graduate supervisors as a massively open online course (MOOC).

The QGS miniMOOC is a hybrid, which means that it was designed to be offered over 6 weeks with the first week including a face-to- face orientation and the rest of the weeks offered fully online. Several factors, considerations, and elements informed the design of the QGS miniMOOC. A conceptual framework for quality supervision emerged from a literature review and three learning theories: constructivism, connectivism, and learning community. Principles of constructivism have been incorporated into the design to promote interactive social learning, collaborative learning, facilitated learning, authentic learning, learner-centered learning, and high quality learning (Huang, 2002). Connectivism theory emphasizes that “instead of knowledge residing only in the mind of an individual, knowledge resides in the distributed manner across a network” (Siemens, 2006, p. 8). Therefore, the design of the miniMOOC emphasizes opportunities for knowledge exchange, discussion, networking, and communication among faculty members from diverse disciplines and fields of study (Alharbi & Jacobsen, 2016). Learning communities in higher education can emerge from the careful design of collaborative learning environments (Tennant, McMullen, & Kaczynski, 2010). The development of a learning community in online faculty development has been found to increase learner success; thus, the miniMOOC was designed to support faculty in engaging in and forming a learning community (Taylor & McQuiggan, 2008). The discussion board in the mini- MOOC was designed to support a learning community among participants while they discussed issues related to graduate supervision and contributed to the ongoing discussions by giving advice and/or examples based on their individual expertise in graduate supervision along with the guidance of the discussion moderators.

Designing and offering professional development opportunities must be strongly tied to the institution’s strategic vision (Camblin & Steger, 2000). The design of the QGS miniMOOC is aligned with two goals of the university’s academic strategic plan: (1) to focus on innovative teaching and learning, and (2) to promote the interinstitutional and interdisciplinary collaboration and internationalization. The university is invested in supporting academic faculty members in developing strong graduate supervision practices and also recognizes excellence in graduate supervision through an established reward structure. The Faculty of Graduate Studies, University of Calgary, has established the “My Supervisor Skills” program to support graduate supervisors with various seminars and workshops throughout the year. Every new academic faculty member must complete an orientation seminar in order to be granted graduate supervision privileges. A faculty member’s supervisory privileges are evaluated every 5 years. Most faculties, including our own Werklund School of Education, offer an annual Distinguished Graduate Supervision Award; the university offers the Teaching and Learning Award for Graduate Supervision each year; the Faculty of Graduate Studies grants GREAT Supervisor Awards each year during Great Supervisors Week; and the Graduate Students Association offers the Excellence in Supervision Award. Established award and recognition structures are an important institutional component in the success of faculty development initiatives, given that the lack of recognition from institutions may discourage faculty from participating in faculty development (Taylor & McQuiggan, 2008). To that end, the support that the university offers for faculty members, along with visible recognition through awards for quality supervision, provided a strong foundation and supportive institutional context for the design and development of the QGS miniMOOC at the University of Calgary.

The design process was also informed by a series of consultations with experts to inform elements of the design. In the analysis phase, questionnaires were sent to all graduate supervisors and graduate program directors at the University of Calgary to gather feedback about the challenges faced by graduate supervisors, the need for faculty development opportunities, and on the desired topics that such faculty development should address (Alharbi & Jacobsen, 2016a). Key themes from the questionnaire data informed the development of topics and learning objectives for the six modules in the QGS miniMOOC. The six learning modules and the subtopics are listed in Table 1.

Each of the six modules was designed to include an overview, learning objectives, learning resources, expert videos, and a discussion board. In the design and development phase, the first author finalized the design of each module with a faculty committee, and set to work developing each of the modules along with the expertise of members of the miniMOOC design team. In addition to resources and guides available from the Faculty of Graduate Studies, a design decision was made to include videos of expert supervisors. One of the challenges at this stage of the research was to attract sufficient funding for the development of the miniMOOC and, in particular, for high quality video production. Acknowledgment is given to the Faculty of Graduate Studies for providing a development grant that enabled the team to work with a professional videographer and editor in the creation of the expert videos for the QGS miniMOOC.

Questions were developed in relation to each of the six module themes and were used as a guide for the expert supervisors during videotaped interviews. The expert supervisors were identified and then invited to participate in the development of videos based on their history of winning a Faculty of Graduate Studies GREAT Supervisor award and/or a Faculty level Distinguished Supervision Award. The expert supervisors who agreed to participate included three males and three females from such disciplines as psychology, biochemistry, medicine, education, and nursing. The videographer set up a studio in a quiet office space to create the conditions necessary for capturing high quality video and sound. Using the prepared questions as a guide, the interviews with each expert supervisor enabled the design team to capture the diverse qualities and practices of excellent graduate supervisors in relation to all six of the miniMOOC topics. Each video interview with an expert supervisor took approximately one hour. The source video was edited such that each of the final expert videos for each inquiry question in a module included perspectives from three different supervisors, and was 2–5 minutes long. The videos designed to be a diverse and rich source of expert supervisor experiences, advice, and wisdom, and also designed to be short enough for several to be watched within a 20–30 minute time frame. Most of the modules had three questions and thus three composite expert videos; the combined video content ranged from 7 minutes total in Module 1 to almost 14 minutes in Module 3 (see Table 4). The selection of the supervisors’ responses to each question was based on two considerations: (1) how informative was the answer, and (2) how diverse the supervisor’s answers were in response to the guided questions. The goal was to include composite expert videos for each module that presented diverse and rich responses to questions such as, what constitutes best practice in graduate supervision, how do you identify students who are struggling, and how do you motivate students? For example, in module three, a composite expert video on “how to establish good communication with your graduate students from the beginning” was posted along with two other composite expert videos related to this module’s topic. Another video on the course home page combines perspectives from all six expert supervisors on what is most satisfying about being a graduate supervisor (Figure 1).

Questions and prompts posted in each module’s discussion forum were designed to be aligned with the questions asked of the expert supervisors in the videos. In Module 5, the discussion board offers case scenarios that the participants were invited to discuss and debate with others. Four of the six expert supervisors who participated in the video interviews, and an associate dean from the Faculty of Graduate Studies, were recruited to moderate the discussion forum for a week to answer questions and participate in the discussion. One discussion moderator requested the opportunity to guide the discussion for 2 weeks in a row after having a very informative experience in the first week. A summary of the design elements and components implemented in the QGS mini- MOOC is in Table 2.

The development process to ready the QGS miniMOOC for initial launch took approximately 4 months, including consultation with the miniMOOC design team at every step of the design, gathering formative feedback on each module as well as the overall design and navigation, altering the design of the modules, consulting and getting more feedback, and making multiple changes and improvements through many iterations. The build using the learning management system Desire to Learn (D2L) took approximately 2 months. The university has limited space for uploading the videos in D2L, so a decision was made to upload the videos at a host website and then include links to the videos in the D2L course shell. As part of the design and development phase, the design and content in the QGS miniMOOC was accessed by three faculty members who were invited to serve as peer reviewers to evaluate the design, give feedback, and suggestions before the QGS miniMOOC was launched with the first group of faculty participants. Each of the reviewers who were recruited were selected for their recognized excellence in online learning and or in graduate supervision. The reviewers provided feedback on the design of the QGS miniMOOC in D2L and on the usability of the course. One of the suggestions provided was to pick brighter colors in the logo and the banner of the course. Another suggestion was to include more direction and guidance for participants on how to navigate the course and to access resources in each of the modules.

The QGS miniMOOC was offered as a closed pilot at University of Calgary in Winter 2017. A link to a recruitment video and e-mail invitation to participate in the QGS miniMOOC was sent to all graduate supervisors at the university. Diverse faculty members from computer science, education, political science, medicine, veterinary medicine, sociology, art, engineering, physics and astronomy, nursing, and physiology and pharmacology elected to take part in the first offering of the QGS mini- MOOC (Alharbi & Jacobsen, 2017). The QGS miniMOOC was launched via three orientation sessions (online and on-campus) for the 23 graduate supervisors from 11 disciplines who initially signed up. The purpose of the orientation was to introduce the faculty members to the QGS miniMOOC, provide a brief demo, and introduce them to the study. Each supervisor was provided with a binder that included an overview of the six modules, the Graduate Supervisor Handbook produced by the Faculty of Graduate Studies, a copy of the miniMOOC orientation slides, and several printed resources from the first two modules. Supervisors were encouraged to print and save resources from the QGS miniMOOC in their binder that they found to be helpful. The demo was focused on showing faculty members how to login and find the QGS miniMOOC in D2L and to navigate the different modules. In response to faculty who signed up from our international branch campus in Qatar, an online seminar was provided because they could not attend in person. Graduate supervisor participants were unanimously very enthusiastic during the orientation about the QGS miniMOOC learning experience and were eager to get started. The instructor, who is the second author, sent the 23 graduate supervisor participants a group e-mail at the beginning of each week to guide faculty members on how to engage in the miniMOOC activities and with the learning resources (see Table 3).

As part of the overall design-based research project, multiple and mixed forms of data were collected during each phase of the research. During the analysis phase, questionnaire data were collected from graduate supervisors at the university and from graduate program directors. To inform the evaluation and reflection phase, data were collected using questionnaires at the beginning of the online course and at the conclusion of the course. Furthermore, data were collected from participants’ and discussion moderators’ activities in the QGS miniMOOC and from interviews with participants, discussion moderators, and the instructor to evaluate the QGS miniMOOC, to learn about what worked and what needs to be changed, and to inform the design principles for faculty development MOOCs. Data were analyzed using thematic analysis and descriptive analysis. Thematic analysis refers to “the process of analyzing data according to commonalities, relationships, and differences across a data set. The word ‘thematic’ relates to the aim of searching for aggregated themes within data” (Gibson & Brown, 2009, p. 127). Themes were generated, organized and compared in order to evaluate the design during the development and implementation of the miniMOOC; however, the focus of this paper is to report on the descriptive data from the participants’ activities in the QGS miniMOOC. Of the 23 faculty participants in the QGS miniMOOC, 14 signed consent forms giving permission to harvest and analyze data about their activities during the miniMOOC, which included evaluating participant: (1) log ins and completion data, (2) access to video segments, and (3) level and quality of participation in the discussion forums. The built-in analytics tool in D2L was used to summarize the quantitative data and the data from the videos were aggregated from the built-in statistics in the host website.

Of the 23 graduate supervisors who signed up for the QGS miniMOOC, a total of 15 participants (65%) completed all six of the modules and were presented with a certificate of completion. Of the participants who completed all six modules, 2 had requested and appeared to have benefited from extra time beyond the 6 weeks to complete the modules. Three participants who finished on time were logging and reading other participants’ posts and watching videos even after the extended formal time for the miniMOOC was over. The elective activity outside of the scheduled 6 weeks for the mini- MOOC is taken as an indication that faculty participants were intrinsically interested in the content, in their peers’ contributions, and benefited from additional access to the online course.

A key element of the overall miniMOOC design was the inclusion of diverse expert supervisors’ experiences, advice, and wisdom through the development of expert videos. In order to indicate the usage patterns by participants and to observe trends, given the expense of developing quality video, the view frequency data on each of the videos was analyzed. Based on the frequency data, the videos that were viewed most often, with 66 views (316.08 viewing minutes), were from Module 1: Introduction to Graduate Supervision. The videos that were viewed the least often were from the Introduction, with a total of 30 views (107.9 viewing minutes). Table 4 provides an overall summary by segment of view frequencies, the number of segments, total time of the videos, total combined views, and the total minutes viewed.

To better understand the faculty members’ engagement in the discussion forums, the following three quantitative measures were used to measure participation in the miniMOOC: (1) the number of threads created, (2) the number of replies to posts, and 3) the number of posts read per discussion forum in the six modules of the QGS miniMOOC (Mackey & Evans, 2011). The total number of “threads created and posts replied” for the first two modules were 46 in Module 1 and 43 for Module 2 and these reflect the second and third highest number of participation events in the discussion forums. Furthermore, the number of posts read in Module 1 and 2 were the highest of all the modules, with 357 posts read in Module 1 and 331 posts read in Module 2. Even though participants viewed the Module 3 videos most often, the number of posts read were the lowest, with only 227 posts read. This trend in participation may indicate that participants were most interested and benefited most from the videos in this module rather than being drawn to posting and responding to each other in the discussion forum. The discussion forum data summarized in Table 5 illustrate the number of threads created, the number of replied to posts, and the number of posts read in all six modules.

In general, the faculty participants actively engaged in the discussion forums and they were observed to be answering each other’s questions and responding to other participants’ posts. Interviews with 7 participants, 4 discussion moderators, and the instructor after the QGS miniMOOC were completed yielded some interesting insights about their experience with this form of faculty development. During interviews, participants reported that they felt more confident in their supervision practices after their participation in the QGS miniMOOC. Participants also reported that the course confirmed some of the ideas that they already had about graduate supervision and that it was a relief for many of them to learn that other supervisors shared some of the same concerns and difficulties with graduate supervision that they had expressed. One participant in the questionnaire stated, “my participation in the miniMOOC not only affirmed areas of strength for me as a supervisor, but it also offered up new strategies I can try to implement with ongoing and new students.” The active discussion about graduate supervision topics among diverse faculty in the discussion forums occurred in a safe and trusted environment, which helped to form a learning community among participants. One participant stated, “I thought it was a really nice online learning community very open and trusting environment where people were comfortable talking about graduate supervision.” Furthermore, participants reported that the flexibility and the accessibility of the QGS miniMOOC was one of the reasons that they had a successful learning experience. One participant reported, “the flexibility afforded by this method beat, hands down, attending an inperson ‘day long workshop,’ for example.” The analysis of participants’ experience affirmed that the QGS miniMOOC was a flexible, accessible and valued learning experience about graduate supervision for faculty members.

Overall, the QGS miniMOOC pilot was found to be a successful learning experience for many of the graduate supervisors who took part, and especially for the 15 who completed all six modules. From the data analysis and the evaluation of the pilot, several recommendations have emerged that inform our next development iteration and phase of implementation:

1. As for timing, the first pilot was offered in March and April, and the final two modules overlapped with the end of semester grading and reporting period. In future offerings, the QGS MOOC will be offered earlier in the semester, such as during the months of October and November, or February and March, or even in Spring or Summer semesters, so that the final two modules do not overlap with the teaching demands that accumulate at the end of each semester.

2. Based on the suggestions from several participants, it was recommended that the design team add a weekly or biweekly synchronous session offered by the instructor or an expert supervisor in a future offering of the QGS MOOC to enhance the development of a learning community from the beginning of the MOOC experience. A synchronous seminar each week or every other week would help to connect participants with each other, and with the experts, for more formal and informal conversations and chats, and would also provide a weekly or biweekly opportunity for the instructor to answer emerging questions from participants in a large group discussion, and also provide an opportunity to summarize and wrap up the topic of the week (Alharbi & Jacobsen, 2017).

3. To amplify and strengthen the learning network, it was recommended that the instructor takes steps to introduce the discussion moderators to participants earlier (live or by video), get the moderators involved in discussions so they can develop a deeper sense of the learning community that participants are forming from the beginning of the MOOC, and become members versus visitors in the learning community (Alharbi & Jacobsen, 2017).

4. To expand the content in the MOOC in subsequent development, it was suggested by participants that the design team add two new modules: one on writing and one on bench or lab science. Participants indicated they would benefit from more guidance on helping graduate students with academic writing across different disciplines, and several also mentioned they would appreciate more guidance on graduate supervision within laboratory settings. For ongoing development more generally, each module will be reviewed and the extant literature and resources available, and the team will expand the rich resources and materials in response to requests for more evidence or another point of view on the module topics.

These findings and recommendations from the pilot of the QGS MOOC can also help to inform instructional designers and educational developers with the design and implementation of successful online faculty development and learning experiences for graduate supervisors.

The first assumption guiding this research turned out to be accurate: MOOCs offer an accessible and flexible and beneficial approach for faculty development in higher education, one that may encourage greater participation and engagement among diverse faculty members. Faculty appreciated and valued the opportunity to engage with other faculty from across academic disciplines to explore, discuss, and share strategies for quality graduate supervision. The second assumption, that a MOOC that focuses on quality graduate supervision can be designed to support the creation of a learning community among diverse faculty members, was also borne out in this design-based research.

The evaluation of the design and development phase in the production of the QGS miniMOOC has resulted in several insights to inform the design team’s ongoing development and future implementation. The design elements that were determined to be relevant and necessary in the development of this online faculty development experience include expert videos, discussion forums, and rich and diverse materials in six modules. The pilot yielded insights on the scaffolding and instructional support necessary to support faculty members to effectively use the Quality Graduate Supervision miniMOOC, from the orientation, to the role of the instructor, the role of discussion moderators, and the timing of the online experience. The pilot yielded recommendations that will inform ongoing development and improve the design for subsequent implementation and evaluation.

It is expected that outcomes and impacts from this design-based research project can inform the design and development of online faculty development and learning opportunities in higher education. It is also expected that findings from this research will be relevant for graduate students, academic faculty across disciplines, senior leadership in higher education, and educational developers.