Four Frameworks for Viewing Blended Learning Cases: Comments and Critique

Called “case studies” in these articles, and even given the lowly name “pilots” by Christensen, each of the studies evidences being set in a sequence of implementations, followed by design of improvements, followed by successive implementations. The most self-aware of these iterative cycles was Christensen's study. She approaches a disciplined comparison of the newly-designed blended study with its predecessor version of the classroom-taught instructional design course. In what is reported as the first cycle, she and her co-teacher document the design changes they made to make the course a blended learning experience consistent with constructivist pedagogy. When a design study appeals to models or theory, even a model rough as the constructivist principles, we would call the design study “principled.” Her designs as reported are clear and principled, though obviously informed much by the craft lore, artfully applied, of the discipline of instructional design methods she teaches. She also reports the design changes made after the first cycle, and the improved results obtained. She concludes with a clear statement consistent with the logic of design studies: “The challenge, then, is to fine-tune the strategies through successive iterations …” Her iterations of design and evaluation deal with such issues as finding the right balance between web and group activities, reducing negative consequences, and improving effectiveness.

One danger of not crediting the power of the J-curve is apparent in her article. Due to negative feedback and marginal value of the first attempt at online peer review, this was abandoned and replaced with face-to-face discussions in class. Yet a key problem in this and other studies like it is that “responding to individual students required a serious investment of my time.” It is known that peer review can be implemented well, can get good results, and can reduce the teacher's grading and feedback time. As a related example, the teacher in the third study had, with experience, cut back on his time in responding to posted messages, relying increasingly on getting the dialogue going well, then only stepping in occasionally. Recognizing that some interventions change habits and practices too much all at once, and that the implementation plan needs to be adjusted to prevent difficulties the next time, perhaps the peer review in Case 1 could have been rehabilitated, rather than pushing it all back on the instructor.

Except possibly for Case 2, all of the cases could have explained themselves better, and made more useful comparisons, by placing themselves in a framework of a sequence of design studies, and recognizing the J-curve effect. Case 5 is an exciting beginning that could become a series of design studies. Like Case 1, it reported on two semesters of implementation (but only the designed changes in semester 2, not the results). Case 5 is incredibly ambitious, redesigning writing courses taken by 30,000 students and using hundreds of graduate instructors. It was the best funded study of the group, with both substantial university and Pew Foundation funds, and with formal evaluation funded specifically, rather than as an “out-of-hide” expectation on the faculty or as a low-budget student project. Interpreted as a J-curve, it is a positive, not a problem that such a small, non-statistically significant dip was experienced the first time in comparison with traditional classes. There was also small evidence of student writing improvement on two aspects of the scoring rubric.

It is good news, not bad, that the instructors were not well motivated nor involved, because introducing features such as the ones they have already proposed for the second semester, it may be possible to get the buy-in and commitment missing the first cycle from the instructors, taking the slope of the J up each semester as improvements are made. Perhaps I detected a note of defensiveness in the discussion of all the changes made to correct problems revealed by the evaluation. This would be normal, after such a large expenditure of leadership and financial capital had produced nothing spectacular.

If a bit of commonly-experienced defensiveness does exist, a forthright grounding of the work as a sequence of design studies (design experiments if the outcome measures become more comparable and valid, and the design changes more principled), takes away any defensiveness. It would then be understood from the first that new implementations, especially with a large group of instructors, are likely to produce no improvement – even a dip from baseline the first cycle, but that the instructional system is now in a form that can be rapidly improved and revised (one of the benefits listed by Osguthorpe and Graham). The greatest improvements related to the J, however, are likely to come as a result of adaptations in the implementation plan – roles (new training and job-aids), rules (new course procedures) and not just in the tools (learning objects, on-line assignment handling, and the like).

Case 5 could also have been improved, had it been conceived from the first as a design study or design experiment. More time and resources would have been expended documenting the baseline – the results and features of the English 115 experience before the change. This is true of every one of the five studies. Some mention was always made of what had happened before the study, but it was clear that some of the cases were well along in a series of implementations and try-outs, followed by design changes, while others (e.g., studies 2 and 5) were more truly the first cycle. The addition of the web-based features that justify the designation “blended learning” is only the latest change by some of these innovative and adventurous teachers. It is clear that this is the case in studies one, three, four. In a sequence of design studies, some of these teachers had already removed some of the glitches and mistakes of first-timers and had developed more seasoned habits. Looking backward and documenting some of these learnings is important in writing up such studies. Take a longer view. What is the baseline? Which semester is really cycle 1? How many semesters can we keep the improvement-oriented sequence going, and how far up the J can we move? What metrics (usually more than one) can we use to make the plotting of the J more rigorous, and comparisons between cycles more attributable to introduced design changes?

Some of the most exciting and promising results in this suite of blended learning studies come from student comments about the great benefit they received from increased dialogue with fellow students. Consider Case 3, a snap-shot partway up the J-curve for an adaptive, innovative instructor. This instructor had used on-line discussion boards before, and “He found individual responses [to student posts] to be especially important during the first month of the semester, when distance learners were becoming familiar with the technology.” In the ECD framework, the first moderating functions are designed to help students “survive” the initial onslaught of technical issues unrelated to the course content, then right after that, to establish a “feeling of belonging” These early moderating functions help students to overcome the fear of rejection or being ignored or laughed at. Once these levels are achieved, a collaborative discourse can accelerate, and can increasingly focus on the content.

Moderating functions are needed to keep the group on track, and to keep the give and take going. This give and take can be very motivating. Later moderating functions, such as “weaving”—putting together comments of others, with excerpts from readings, into a higher synthesis helpful to everyone—can lead to high levels of real intellectual engagement. This is exhilarating for the teacher, and for the students alike. In Case 3, “the instructor was able to see that this objective [learners becoming more conscientious scholars] was being met. He was especially pleased when learners would include links to articles or include valuable references he had not previously accessed.” High-level synthesizing and weaving comments in student posts are a new form of evaluation data useful in detecting higher levels of intellectual engagement.

An understanding of the the use of and response to moderating functions by instructors and students alike in the use and response to moderating functions can inform implementations of cases other than Case 3. In Case 2, teaching and applying an understanding of ECD and moderating functions could benefit the dynamics of the blended class significantly. It could benefit interactions locally and collaboration with the distance students. Encouragement through moderating functions would help all students feel more involved and would facilitate the effectiveness of the chat sessions as well. ECD through asynchrounous seminars could be added in a later cycle to further enhance collaborative discourse for all. Lessons from ECD apply in Case 4 as well. Here, the students engage in group discussions, with an overt goal to help them operate at higher levels of critical thinking—the upper end of the moderating function continuum—promoting intellectual engagement. Moderating functions can be applied in oral situations as well as in computer text dialogues. Also, in future cycles of implementation, the instructors in the accounting classes might consider the use of asynchronous computer conferences. If not feasible in the 100 student classes, perhaps in later, smaller classes where intellectual engagement is the clear goal.

Case 5 offers some of the greatest benefits potential benefit attainable through teaching instructors to use computer conferences and peer reviews to teach the writing process. In cycle 1, instructors traded off office hours, direct instruction, and other matters for 110 minutes, on average, of email. If they knew the ECD process, and the high levels of intellectually engage application of writing principles attainable thereby, it might bode well for increasing their own intellectual engagement and feeling of accomplishment.

A few comments reported from students interviewed or polled in these studies showed individuals reflecting the need for accommodation to their differences. In Case 4, one student said “I just wish that on some of the easy points, you could click through a little faster.” This is the kind of comment that would be made by a transforming learner, the most eager, self-confident, and autonomous of the four levels of the learning orientation construct. (Performing learners are not as demanding of learner options adaptive to their own style and pace, and conforming learners want to be told step by step what to do and how they are doing along the way. Resistant learners [not many in college] dare you to make them learn.) Another student in Case 4 stated: “I thought the step-by-step process on the homework assignment allowed me to check my own work step-by-step as well.” This is a comment consistent with a conforming or low-performing orientation, one who likes a careful, step by step treatment and was well accommodated.

It is possible to identify learning orientation by means of a short on-line questionnaire, and it behooves designers to make provisions for the kinds of instructional support favored by learners of different orientations. This includes navigational options to move rapidly, using self-selection of sequence at one extreme, and slow, careful, step-by-step guidance and examples at the other. This is relatied to the personal agency issue identified by Osuthorpe and Graham in the lead article.

One point of critique about the generally excellent Case 4: it may obscure more than it reveals to say that self-study students going through worked examples were “getting answers to their questions.” It is true that questions arise in the minds of students as they click through these examples, and that some of their confusions and impasses may be overcome during this process. Worked examples are more instructionally complete than words alone. This is not the same as getting answers to oral questions or text questions asked in English, any more than poking the buttons on a telephone and getting branched computervoice answers all of the questions we have called some company to ask about.

VCD is based on a series of cycles of design-implement and evaluate-revise. It is a method both to guide the design of measurement instruments, instruction, and implementation plans, and to seek evidence and interpretations of consequences to guide the revisions and improvements prior to the next cycle or implementation. VCD assumes a design study or design experiment way of conceiving the work of implementing and improving the system—blended learning system in this instance—and it organizes a set of methods appropriate to improving the validity of each system and subsystem in the entire blended system.

The data is organized in terms of three main categories of validity, and is documented as a “validity argument”. Validity is never fully established, since a blended system keeps changing with the times, with teachers, with technology, content changes and the like, and it cannot be completely finished all at once even when a system stays stable over a number of cycles (Cronbach, 1988, Bunderson, 2002). VCD is a recent term and evolving method. It builds on the work of major validity theorists and applies the concept of the unified validity model in a new domain, that of blended learning and other systematically designed educational interventions. It is a framework which enables a critique of studies like the five in this issue, because it both connects and elucidates issues the authors are dealing with, and also suggests those that have not been identified yet in each study.

There are issues of user-centered design, inherent construct validity, external evidence and consequences that should be considered in each serious blended learning implementation that has the commitment and support to keep improving over cycles. These issues play out in different ways in each of several areas of design-for-improvement:

1. better measurement instruments to reveal what students know and can do at progressive levels of growth, and to reveal aspects about the blended learning system itself and how it functions;

2. the design of instruction for each of the several main paths of learning and growth provided by the blended system, and for each level from early to advanced;

3. the design of adaptations to individual differences; and

4. the design of implementation plans that, regardless of no improvement or a delay before improvements are manifested, can guide the system to ever higher levels by reducing first-time mistakes, training better, managing better, and incorporating system improvements better.

The idea is to show over cycles a good, steeply rising J-curve. Table 1 summarizes these functions of VCD.

Many considerations are a part of category I of VCD. There are established methods and standards to guide in documenting evidence for success in user-centered design.” Is the system appealing, easy to use, cost-efficient, and does it portray adequately to the user that which the user might value most? Does it portray a vision of what the user can expect to accomplish? Case 1 showed great sophistication in the design of the user interface for the course in instructional design. The author used constructivist principles and hard work on her part to adapt the student project and feedback to individual interests and values while working on it. Thus she did a very good job in cell 4. She may not yet have systematized 7 and 10, but gave evidence that instructor management is effective in these areas. Studies 2, 3, and 5 relied on the user interface provided by Centra and Blackboard, but some of the student comments suggested desired improvements. Such changes are hard to obtain from outside vendors. Case 4 showed evidence of efforts to develop usable and effective on-line study materials. In future studies perhaps these can be improved by attention to 7, 1, and 10 in addition to 4. No blended learning implementation is totally limited by its user interface, even when these are provided by a vendor. Because the teacher's voice and presence is available, there is much the teacher can do to convey the values and expectations that might connect with those the students hold. The teachers can also seek to understand better what these are, and to provide training and moderating guidance, especially during the first stages, as the teacher in Case 3 did, so that students will not become frustrated before beginning to see some of the values they seek.

Studies 4 and 5 overtly dealt with the issues of time, cost, and efficiency that fall under category I of VCD. Their designs and improvements are providing increased understanding of these issues and how to manage them.

This is a term that relates to the link between the constructed ideas of expertise students should be able to attain as a result of the instruction, and the measures supposed to assess what the students actually know and can do it. It contains three aspects, content coverage and appropriateness, attention to the thinking processes and how they develop, and attention to the link between score scales and what they should be measuring. We construct our ideas of expertise from previous experience by listing content topics, writing performance objectives, and developing assigned tasks. Some tasks appear in tests or rated work products to assess student progress others provide practice. It is easy to see that this aspect of validity is at the heart of the measurement system—does it measure the outcomes the designers and the students really value? It has not been common in instructional design to recognize the importance of inherent construct validity in the guidance, examples, tests, etc. we produce to teach. Yet practice does not make perfect if one practices the wrong thing, nor does instruction help if it teaches toward something other than valued outcomes that really characterize true expertise in a particular learning domain.

The teachers in Case 4 showed a nuanced understanding of the thinking process aspect of inherent construct validity. In their Figure 1, they showed a model of two levels of thinking processes, the lower dealing with declarative and procedural knowledge about the tools and technical information, the higher dealing with decision-making abilities in applying the tools. They showed how their design of the blended learning environment accomplished the first objective through individual study of prepared multimedia presentations, and they increased discussion time by freeing up classroom time to model and guide students orally in the higher objective. Perhaps they have measurements to show their success in accomplishing each of these levels; perhaps, like most academic instruction, the measures of student outcomes are less formal. They could find out in a more rigorous manner by implementing on-line the objective parts of the tests, and entering ratings of work-products on-line to tighten the evaluation-improvement loop each semester. In this way they might attain even better results and more fully capture the power of Ease of Revision cited as a key benefit in the lead-off article.

The designers-evaluators in Case 5 showed evidence of good domain modeling—a prior step in designing construct-valid measures. They discussed the writing process and how their system, and good instructors, could model it and then provide feedback as students attempted to practice this expertise through their four papers. Unfortunately, they also presented evidence of the wide disparity of grading practices among instructors, a failure in applying a common model of domain expertise and a serious problem of fairness to the students and validity of the grades resulting from this process. It is clearly a problem not only in the validity of the measures, but also in the validity of any instruction geared to improve performance in developing these written work products. To attain higher levels of construct validity, it is surely necessary to give students the opportunity to develop work products that integrate understanding and reveal different levels of expertise. In judging these work products, it has never been shown that valid and reliable grading can be accomplished using only one reader, (although it can be improved by substantial and consistent training of the readers).

Since it is not likely that the English department will pay for two or more readings per essay, the practice in national and state high-states assessment, we ought to look to peer grading of at least some of the work products. It has been shown that peer grading not only helps to manage this serious work problem for instructors, but is itself a powerfully effective learning method. Moving up from the bottom of the J-curve in both studies 1 and 5, without presenting so much work that most instructors will not do it, can surely be helped by sophisticated, iteratively improved, peer grading and collaborative discourse about it.

The domains in Studies 2 and 3 were not clearly articulated, so it is not possible to judge from these papers the extent to which the construct validity of the measures used had been a design issue. In Case 3, the evaluators did observe that the instructor valued thorough scholarship and citation, and looked for it in posts in the computer conference. Learning to evaluate the depth of computer conference postings is surely a fertile area for improved assessment of higher levels of engaged thinking in learning domains such as those found in the two religion classes.

There are three aspects to category III of VCD. Generalizability evidence is the first of these. Do the blended learning systems generalize to different sets of learners, instructors, and different learning tasks? Generally, studies that compare the effects of an intervention of different gender, ethnic, age, and other groups are conducted to provide evidence of generalizability. Such investigations are largely in the future for the blended classes in these five studies. Case 5 is an exception, an unusual test-bed for investigating the generalizability of a new blended course design to many different student groups under the tutelage of many different instructors. While evaluation as a separately funded activity is not likely to continue at the level reported in Case 5, teams can be set up to “build in” evaluation, especially through improved measurement of the students' knowledge and performance abilities in writing their four essays. A long series of cycles in this J-curve is required to achieve this, because of the large number of teachers and students. Yet the promise is great, because good people committed to improving a system under their stewardship can produce amazing improvements over time if given the chance, and when provided with metrics to show when progress is made.

There are many other forms of external evidence that can show the extent to which a blended learning implementation is valuable and effective. One form of external evidence is particularly germane to blended learning studies. It has to do with what has been called Kirkpatrick (1994) levels 1-4. Level one is reported in these studies—reaction sheets and interviews about how the students liked it. Level 2, how they did on the objectives of each course, was reported in several of the studies. It requires attention to good within-course measurement as discussed under II above. Level 3 deals with results beyond the course—questions of whether or not the designers in Case 1 continue to design and develop ever-better projects after the class, whether the religion students recalled and continued to use the information and the scholarly skills treated in the classes, whether the finance students continued to use and improve accounting skills appropriately, and whether the English 115 students continued to use and improve their writing skills.

Greater external evidence that the blended learning courses are attaining their goals at levels 2 and 3 are in the future. Paradoxically, there is some evidence at Kirkpatrick level 4. This level deals with the extent to which increased effectiveness of these courses has an impact on mission attainment and the bottom line at BYU. The evidence provided in two of these five studies that reached out to off-campus learners is highly germane to BYU's goals to reach far more students than can be accommodated on-campus. There was also some evidence of increased cost-effectiveness. Part of the enthusiasm in the write-up of these studies relates to this promising connection to larger institutional goals.

VCD Category III also includes concern for consequences. Positive consequences can be argued using a variety of evidence, much of it related to the Kirkpatrick levels mentioned in the previous paragraph. Continued vigilance for negative consequences is also a mandate of Category III. The authors were thorough in documenting some unanticipated negative findings, and in looking ahead to how to correct each problem.

Mention has been made throughout of the importance of certain of the six goals, or benefits, of blended learning outlined in the lead-off article. These are certainly important benefits to be sought through blended learning. These six are centered most precisely in the column of Table 1 dealing with the instructional system. They overlap in minor ways into other columns, and fit best in certain rows. For example, social interaction and personal agency are Values, (as well as methods), and fit in row I. Social interaction and agency are more or less attractive to students from the outset who are looking at a blended learning alternative. Personal agency interacts with individual differences (column 3 of Table 1). Conforming learners may be repelled by too complex a requirement to make a lot of decisions and choices about their own learning. Transforming learners may demand it or avoid the course altogether. Social interaction seems greatly enhanced by blended learning, and is most critical and most difficult when distance students are added to pre-existing classes. In these cases, social interaction through computer conferences and text chat was shown to be of great value. In conclusion, the four frameworks for viewing blended learning cases, along with Table 1, are offered as an expansion and elaboration on the six goals or benefits of blended learning offered in the lead-off article. Within each of the four, guidance is offered for both design and evaluation, linked in repeating cycles of improvement.