Online Learning Design and Implementation Models: A Model Validation Study Using Expert Instructional Designers

Online learning continues to grow at unprecedented rates. In higher education over 7.1 million students took a course in 2013, which represented over 33.5%of the total student population (Allen & Seaman, 2014). K– 12 is growing at similar rates, with over 275,000 students enrolled in full time virtual schools in 2011–2012 and over 2 million participating in at least one online course (Allen & Seaman, 2014). Corporations are experiencing growth as well, with technology-based learning at 37.30%and online learning at 24.77% (Miller, 2012).

Despite this unprecedented growth, there is little research on model formation and model validation for online learning (Richey & Klein, 2007; Richey, Klein, & Tracey, 2011). Two research-based models were created: eSUCCESS (Armstrong, 2007, 2008, 2016b) and R2D2/C3PO (Armstrong, 2014, 2016a) to help project managers, instructional designers, and developers design and implement online learning solutions. Both of these models were created based on research findings using qualitative case study and design and development case study guidelines. Both models have a strong focus on culture, learning, and technology, and the change management strategies necessary for successful design and implementation of online learning in different contexts such as, K–12, higher education, healthcare, military, and corporate organizations. The specific problem that was the focus of this study is validation of the two research based models.

The purpose of this descriptive qualitative study was to describe the process of model creation for both models and to systematically validate the two models using a survey and the NGT with expert instructional designers. For the purpose of this study, the NGT was based on Potter, Gordon, and Hamer’s (2004) adaptation of the technique. Participants were instructional designers and instructional technologists, who are experts in online learning across industries. Inclusion criteria included more than 5 years of instructional design/instructional technology experience and a doctoral degree in instructional design, instructional technology, or related field.

The research questions were focused on validating the two models based on the experiences of expert instructional designers and align with Yin’s (2014) assertions that case study questions should be “how” and “why” questions and Thomas’s (1983) expert requirements that the questions be clear and stimulating for the participants.

The methodology was qualitative and the design descriptive case study. Data collection techniques included document review of the models, expert survey responses, and a focus group interview using the NGT technique. Six participants were asked to review materials on the models and provide comments and complete a survey on the models premeeting. Five agreed to participate. Following the individual model review, three of the five participants attended a 90-minute NGT focus group discussion, responding to and commenting on each element in the models. The focus group session was professionally facilitated, digitally recorded, and transcribed verbatim.

Jones and Hunter (1995) emphasized the importance of recruiting participants who are subject matter experts on the topic. Experienced instructional designers having at least 5 years instructional design experience and an advanced degree in instructional design, instructional technology, educational technology, or an equivalent degree were recruited. A purposive, convenience sample (Miles & Huberman, 1954) was used based on more than 25 years of networking experience with instructional designers who met the inclusion requirement.

Several instruments were employed during the data collection process: (1) predata collection questionnaire, (2) prefocus group model evaluation and survey, and a (3) focus group protocol based on the NGT. The model evaluation, model survey, and focus group data were used for convergence of data following Yin’s (2014) call for triangulation to ensure the validity of the data. The Model Evaluation Surveys required participants to write down their own ideas that come to mind as they are reviewed each of the models. The Model Evaluation Survey also required the participants to rank the elements of the models using a Likert Scale.

The NGT is a five-step process whereby participants meet one another, generate ideas based on their prework review, share ideas with the group, discuss ideas, and vote and rank items based on original questions about the models. Delbecq and van de Ven (1971) stressed the importance of using an expert in the topic area during the NGT session and Gallagher, Hares, Spencer, Bradshaw, and Webb (1993) stressed that the facilitators must be highly skilled in group facilitation techniques. Both facilitators were experts on the topic, having worked on the design and development of the models and both are trained and endorsed Dale Carnegie trainer/facilitators. The focus group was held online using a virtual classroom technology, GoToMeeting. Prior to the NGT focus group and following receipt of Informed Consent and Northcentral University IRB approval, participants received an introduction to the NGT protocol and several published articles that explained the models. Participants were then asked to rate each item in the models on the Model Evaluation Survey using a Likert scale from 1–4 to rank the importance of each element in the model, with 1 as unimportant, 2 of little importance, 3 important, and 4 very important. Space was available for comments and suggestions. Estimated time was 90 minutes. Following receipt of the surveys, an NCT focus group session was scheduled. Each of the steps in the NCT protocol was followed:

Step 1.Introduction and explanation: A focus group session was held online using the GoToMeeting virtual classroom technology. The session began with an introduction and explanation of the purpose, a review of the agenda (appendix C), and an overview of the procedures for the meeting (5 minutes).

Step 2.Silent generation of ideas: The second step for each model was quiet time for silent generation of ideas. Participants were asked to reflect on each model separately and generate ideas for what components that would add to each element of the model and to the model itself and to write these ideas on a sheet of paper (10 minutes).

Step 3.Sharing ideas: Participants were asked to share their ideas based on their review of the materials in Step 2 and prior to the NCT session. Brookfield and Preskill’s (1999) Circle of Voices protocol (appendix D) was used so that each person could further explain their ideas, initially without interruption. In this step, no debate was allowed and everyone had an opportunity to make an equal contribution to the discussion (3–5 minutes per participant). Total time 15 minutes.

Step 4. Group discussion: Continuing with the Circle of Voices protocol, after each participant had shared their individual ideas, group discussion was opened up and cross talk allowed (15 minutes).

Step 5. Voting and ranking: Facilitators reviewed suggestions for enhancement to each model and presented the suggestions to the group. Participants voted on each suggestion and the results are in the results and findings. Each participant later e-mailed their silent generation of ideas sheets back to the facilitators.

Data analysis included the use of descriptive statistics generated in EXCEL from the survey. Results were calculated for the average, mean, mode, and standard deviation of each element of the models. Themes were generated from the following: (1) comment section of the surveys, (2) NCT focus group discussion session, and (3) silent generation of ideas sheets. Each data collection vehicle was analyzed using NVivo software, which was used as a centralized database for all study materials. A preliminary list of start codes based on the elements of the model was used to create parent and child nodes and start the data analysis. The preliminary list of start codes, transferred to parent and child nodes, was supplemented with new emerging themes from the analysis. Finally, in the data analysis are the results of the voting and ranking at the NGT discussion session. Results of the analysis are presented as a narrative and as a matrix summary to show the level of support for each of the model components. These data will be used to update the models.

The goal of this study was to use expert subject matter experts to validate two researchbased models. The first model was the R2D2/C3PO model based on Bonk and Zhang’s (2006) R2D2 model that focused on constructivist strategies for developing online instruction. Armstrong (2014, 2016a) expanded this model during design and development case study research using a blend of synchronous and asynchronous delivery in higher education classes (Armstrong & Thornton, 2012). Five components were added to Bonk and Zhang’s original model along with tools, techniques, strategies, and activities for each component of the model.

Table 1 shows the R2D2/C3PO model as an expansion of the original Bonk and Zhang (2006) R2D2 model. Each component is described, followed by instructional strategies and activities that can be used for each component. Finally, synchronous tools to support each of the learning activities and instructional strategies are described.

Following Christensen’s (2006) stages of model building, Armstrong (2007, 2008, 2016b) observed chief learning officers of some of the largest corporations in the world and interviewed these chief learning officers as they rolled out large-scale learning management systems and online web-based learning courses for several hundred thousand employees. Following observation and interviews, data was categorized into themes and associations and connections reported (Armstrong, 2007, 2008). The analysis revealed five major categories and 30 indicators that ultimately became eSUCCESS. Reducing the data further, eight tenets were identified that embodied how chief learning officers described the necessary elements for successful implementation of an eLearning system. Table 2 shows the tenets and their descriptions.

To meet the NGT requirement that evaluators were experts in the field, a predata collection questionnaire was used to ensure each participant met the study inclusion criteria. Inclusion criteria included more than 5 years of instructional design/instructional technology experience and a doctoral degree in instructional design, instructional technology, or related field. Table 3 shows participant demographics. All experts had at least 15 years instructional design experience, with three having over 20 years. All experts had doctoral degrees in instructional design, instructional technology, or a related field. All experts worked in field where they applied instructional design and instructional technologies. Table 3 details participant demographics across the inclusion criteria for the study.

Table 4 shows the results from participant evaluations of the R2D2/C3PO Model Evaluation Survey (Appendix A). Each participant was asked to rank each component from 1–4, with 1 as unimportant, 2 of little importance, 3 important, and 4 very important. All components of the model received at least a 3, important, with the exception of 1b (create participant manuals for live synchronous events), 2b (reflection time allocated during synchronous event), 2e (focus group discussions), and 2f (online role play).

Table 5 shows the results from participant evaluations of the eSUCCESS Model Evaluation Survey (Appendix B). Each participant was asked to rank each component from 1–4, with 1 as unimportant, 2 of little importance, 3 important, and 4 very important. All components of the model received at least a 3, important, with the exception of 3h (threat of job loss) and 7d (traditional classroom).

Results from the NGT focus group yielded important addition suggestions for the two models, which are presented in this section. The suggestions for the R2D2/C3PO follow and each is listed under the appropriate component in Table 6 in bold.

Read/Listening. Adding “live authors” to the synchronous expert lectures was suggested based on experience from several of the experts.

Reflect/Writing/Sharing. Recommendations were made to enhance the reflection time by including “silent time” for reflection along with questions that encourage deep thought, such as “what was learned,” “how it was learned,” and “how the learning can be applied.”

Display. Recommendations to enhance the display component included additional technologies such as virtual worlds and mobile technology, along with detailed demonstration techniques such as process steps, concept maps, logic steps, and using real-world work examples. There was a recommendation for providing instructor live video-based feedback of work products.

Doing. Recommendation for adding adaptive learning and gaming to enhance interactivity and student engagement were suggested.

Conviviality. Adding to the model a focus on support for the instructor/facilitator by focused attention on instructor/facilitator workload pre- and postlearning event, as well as during the learning event.

Planning and Organization. The importance of a component for both formative and summative assessment was emphasized, which can be readily accomplished through use of polls and observation. Adding a Facilitator/Instructor Manual that aligns with the Participant Manual would enhance the model and allow for greater consistency in delivery.

The suggestions for the eSUCCESS model follow and each is listed under the appropriate component in Table 7 in bold.

Tenet 1: eXecutive Sponsorship. Adding mission to the description of vision and values was considered important.

Tenet 2: Support From the Organization. More specificity around the concept of “built into the organization,” should be added including: needs assessment, continuous improvement in updated courses, and focus on learning transfer.

Tenet 3: Understand and Motivate the Learner. Going beyond a performance plan to a development plan that focused on motivation, training, remediation, and badges was recommended. Offering badges was recommended for fostering intrinsic motivation.

Tenet 4: Culture Fosters Learning. Making sure that all leaders have leadership training and are required to attend various trainings with the “rank and file.” Emphasis on “futuring” and visioning and change management strategies should be included.

Tenet 6: Evaluate and Assess. Add Philips Level 5, Return on Investment (ROI), Return on Training (ROT), and continuous program evaluation and continuous needs assessments.

Tenet 7: Structure of Program is Engaging, Interactive, and Blended. Add mobile technologies.

Tenet 8: Simulate the Work Environment and Work Tasks. Add guided practice with feedback, Ted Talks, YouTube videos, gaming, and virtual worlds.

Many models in instructional design and instructional technology and media are developed based on experience in practice and/or hearsay, rather than on rigorous empirical study (Richey et al., 2011). Richey and Klein (2007) explained that these models should be validated using design and development research. Both the R2D2/C3PO model and the eSUCCESS model emerged from rigorous research. The goal of this study was to further validate the models by intensive review by instructional designers and instructional technologists who had extensive and intensive experience in the field as well as doctoral degrees in the field. The models can serve as a guide for practitioners as they seek to develop online learning programs. The expert validation strengthens the models and their use in practice.

Expert validation confirmed that the major components of the model were predominately important or very important, with a small fraction being of little importance. While the experts did not add major elements to either model, minor enhancements were made to most components of both the R2D2/C3PO and eSUCCESS models and will further augment the models and provide stronger frameworks for professionals to use as a guide for implementing online learning and synchronous online learning in their organizations. Two data collection techniques, expert Survey and expert NGT focus group, were primary in providing trustworthiness, transferability, credibility, confirmability, and dependability to the results and findings.

The field of instructional design and instructional technology is relatively new and growing rapidly as technology is changing at an ever-increasing rate. Often models, frameworks, and processes are created based on experience in practice or limited research. Future research using a quantitative correlational analysis of the different components of both models would further validate the major categories and components of each model, thereby, enhancing their importance to researchers and practitioners alike. Applying the models in practice and reporting on the findings in a design and development model research study would be another way to extend the research. Additions to model components or categories from that research would serve to further enhance the instructional design knowledge base.

Please rate each element in the R2D2/C3PO model as to its importance in the model on a scale of 1 to 4. 1 = unimportant; 2 = of little importance; 3 = important; 4 = very important.

Comments. Please use the space below to comment on any of the components of the model. Add any suggestions you have for elements that should be added to the model.

Please rate each element in the ESUCCESS Framework as to its importance in the model on a scale of 1 to 4. 1 = unimportant; 2 = of little importance; 3 = important; 4 = very important.

Comments. Please use the space below to comment on any of the components of the model. Add any suggestions you have for elements that should be added to the model.

Welcome and Overview (2 minutes)

Process for each model

• Step 1. Review of Model Components and Silent Generation of question(s) to be addressed. What components might you add to each element in the model? And, to the model itself? (10 minutes)

• Step 2. Sharing Ideas. Each participant shares their ideas about additions. No debate at this stage (2–3 minutes each). (10 minutes)

• Step 3. Group Discussion. (20 minutes)

• Step 4. Voting. (3) minutes

Circle of Voices

• Individuals reflect on the discussion topic (1–3 minutes)

• Participants go around the circle in order—each person has up to 1 minute of uninterrupted air time to give their viewpoint on the topic. No interruptions are allowed.

• Move into free discussion with the ground rule that every comment offered must somehow refer back to a comment made by someone else in the opening circle of voices. This need NOT be agreement—it can be a disagreement, a question, an elaboration or extension, an illustration, and so on.

Retrieved from http://www.stephenbrookfield.com/Dr._Stephen_D._Brookfield/Workshop_Materials.html