CASE 5: Blended Teaching and Learning in a First-Year Composition Course

The Board of Trustees of BYU, a large, privately funded “teaching university,” have encouraged the institution to find ways to reach more students without increasing its enrollment. The current University president Dr. Merrill Bateman (1998), has supported faculty teams interested in creating blended courses that had the potential to (1) create efficiencies by substituting capital for labor and (2) improve teaching and learning through strategic uses of technology. Because it is a core general education course and affects most of the first-year students on campus, English 115 was one course selected for our redesign efforts.

In the general education program, first-year writing is presented to students as one of the “languages of learning,” introducing students to effective writing processes, processes of critical reading, library research, information literacy, and knowledge of academic genres and conventions. The course enrolls approximately 3,300 students each academic year in 87 sections per semester with 20 students per section. The course traditionally meets three hours each week and is taught primarily by graduate instructors in the English department MA program, who have full responsibility for the course. Faculty, administrators, and instructional designers were convinced that redesigning this course to be a blended learning environment could introduce efficiencies both in terms of classroom space and teaching time, and could improve the quality of student writing through increased personal interaction. The authors describe the process of redesign and results from Winter 2002 implementation.

The redesign of English 115 was a collaborative effort involving faculty in the Department of English and a team of instructional designers, programmers, editors, and artists from BYU's Center for Instructional Design. This redesign of English 115 was funded by Brigham Young University and by a Pew grant from the Center for Academic Transformation. The process of redesign involved refining course objectives and assessment strategies and attempting to capture the best practices of English 115 instructors. The design team then tried to emulate these best practices in 25 multimedia lessons that would follow the major units and assignments in the course. These multimedia lessons include video, interactive learning objects, brief assessments with feedback, sample essays, and brief readings that require response. Two experienced teachers of the course piloted these multimedia lessons Fall semester 2001 in preparation for a formal evaluation of the course during Winter Semester, 2002.

To measure the effectiveness of the blended learning environment for English 115,the authors conducted an extensive evaluation during Winter Semester, 2002 comparing the blended with the traditional sections of the course. This evaluation included pre- and post-course student surveys, instructor time surveys, instructor interviews, feedback from student focus groups, completion rates and grades, and student writing samples. These data were analyzed together to create a case study of blended teaching and learning within this learning environment. From this case study the authors have developed basic principles of teaching and learning within the blended environment.

Traditionally, English 115 meets three times a week for 50 minutes each session. The enrollment cap for traditional classes is 21 students, and during the semester under evaluation, two of the traditional sections studied had 20 students enrolled, and one of them had 17 students. Class time focuses on assigned readings from Writer as Work, the textbook for the course. In addition to traditional lecture, instructors are strongly encouraged to devote a significant amount of class time to interactive and workshop activities. Each course requires the same four major papers: a personal narrative, textual analysis, research paper, and group negotiation paper. Graduate students serve as the instructors for the course. Although they are supervised by the Composition Director and his assistants, graduate instructors are autonomous, in that they design their own in-class activities, focus the specific parameters of each assignment, and grade all assignments on their own.

The blended course, meets only one time per week for 50 minutes. The enrollment cap for these sections is also 21 students. Of the courses under review, two of them enrolled 19 students, and one enrolled 17 students. Class time focuses on assigned readings from the online course module, which is modeled after the textbook, Writers at Work. Online students are not required to purchase a printed copy of this book. In addition to traditional lecture, instructors for the blended section are also encouraged to design interactive and workshop activities. However, because some of the course is online, there is a strong temptation to devote time to administration and technological questions. The courses are taught by graduate students whose responsibilities and assignments are comparable to those of other English 115 graduate instructors.

When face-to-face instructional time is replaced with blended learning activities in a blended course, some become concerned that faculty will be less satisfied with learning in this environment and that teaching will require more time. These are both important questions that warrant empirical investigation.

To measure instructor time, the authors prepared and distributed a weekly time-log survey to the instructors of the traditional and blended courses. This survey asked the instructors to record how much time they spent each week in the following categories:

1. preparation,

2. instruction,

3. grading,

4. office hours,

5. email, and

6. teacher development.

During our instructor interviews, the authors also asked how they managed their time. The results of these analyses are shown in Table 1.

Our analysis showed an overall 25% time savings for the blended course versus the traditional course. Generally, these time differences occurred where the authors expected: in instruction (because the blended class met only once a week), and in office hours (because instructors in the blended sections did not hold regular office hours). Although instructors in the blended course spent more time emailing their students, the reduction of office hours compensated for the time spent emailing. Instructors said they used email much like office hours to answer student questions about assignments and to provide feedback on student work. Instructors in traditional courses said students rarely visited them during office hours but preferred instead to meet during scheduled conferences. Although instructors in the blended course often used email, they did not take full advantage of asynchronous discussion boards for student-instructor and student-student communication. The three instructors in the blended course did not feel comfortable using asynchronous discussion because they felt overwhelmed with learning the new technology. During a second implementation of the course during Fall Semester 2002, instructors made much more extensive use of the discussion board.

From our interviews with the instructors in the blended course the authors learned that they spent “preparation time” learning to use Blackboard, fixing technical problems, posting quizzes, and creating their Blackboard site. In other words, their preparation time was spent administering the course, rather than preparing lesson materials. In addition, the authors found that these instructors spent much of their valuable class time answering administrative questions and solving technical problems. To ameliorate these situations during our second implementation, the authors developed more extensive training that focused on using the Blackboard course-delivery system and taught the instructors to effectively implement the course modules. Areas for improvement included how to best use asynchronous conferencing software, how to best use weekly class time, and how to best structure writing conferences.

Generally, instructors were satisfied with the flexibility and convenience of the blended course. Because these instructors are also graduate students, they appreciated the fact that they could focus more energy on their own research. When asked whether he was satisfied with the teaching experience, one instructor said, “in a way I am satisfied because I am not as emotionally involved. I have more time to focus on my own research.” Yet this statement reveals the major weakness the authors found with the experience of teaching in the blended environment: Instructors lacked a sense of ownership and pride in their students' success.

This lack of ownership was due to the fact that the instructors did not design any of the blended course activities and did not use their face-to-face time effectively. (Instructors were involved extensively in the original design of the modules, but by the time the modules were available, these instructors had graduated.) Often, instructors felt as though they were TA's or graders, rather than teachers. One instructor said, “I never prepare class activities or content. I spend most of my time troubleshooting.” In preparation for our second implementation Fall 2002,the authors trained instructors to use their class time for instruction, rather than for “troubleshooting.” The authors offered them tips for moving all of the administrative functions of the course, such as quizzes, assignment descriptions, and technical support, to Blackboard. The authors also provided them with a handbook that gave advice for designing their own instruction-ally-sound blended activities for each unit and lesson in the course. This combination of technical and instructional training allowed instructors during the fall implementation to exercise their abilities as instructors, rather than as mere graders.

The question for many involved in this project was how the course redesign influenced students' satisfaction with their experience in the course, and whether the redesigned course produced better writers. In short, did the authors sacrifice quality of learning and student satisfaction in the blended learning environment?

Through the surveys and focus groups, the authors measured overall student satisfaction in the blended versus the traditional course. From our analysis, the students rated the quality of the traditional and blended version of the course almost identically. The response rate to both the pre- and post-course surveys was fairly high. A total of 69% of students enrolled in the blended course responded to the pre-course survey, while 100% of students in the traditional course responded. For the post-course survey, 64% of students enrolled in the blended course responded, compared to 87% of students in the traditional course. Table 2 shows that 80% of both students enrolled in the traditional course and students enrolled in the blended course felt the overall quality of their experience was good to excellent.

We asked additional questions about the quality of the assignments and the clarity of the course objectives. Students rated both assignments and objectives lower in the blended version of the course. Of students enrolled in the traditional course, 6% were dissatisfied with the assignments and objectives. In the blended course, 21% of students were dissatisfied with assignments, and 18% were unhappy with the clarity of objectives.

We found that instructors assigned all the blended reading and assessments from the blended course modules. These were assignments that they did not write, probably did not understand, and for which they felt little ownership. According to one instructor, it was “difficult for a student to see the purpose of an assignment when that purpose is unclear to the instructor.” There are a few ways to mitigate this: instructors could be free to design all of their own activities in the blended course, or they could receive a more in-depth introduction to the course material, so that they know what is there and what they are assigning. The best solution to this problem likely requires a balance of these two approaches, involving current instructors in the continuing development and refinement of learning objects, as well as introducing instructors to learning objects that have already been developed.

Students were generally very satisfied with the level and quality of the student/instructor interaction. When the authors asked whether the students were satisfied with their level of instructor interaction, 40% of traditional students said they were satisfied, compared to 48% of online students. Although students in the blended course seem fairly satisfied with their level on instructor interaction, they made it clear that face-to-face instructor conferences were a must. Students said that a purely online, self-paced version of the course would be very difficult. They were pleased with how accessible their instructors were, letting students call them at home and responding to their emails promptly. Indeed, students in both versions of the course rated the level of instructor feedback highly: 92% of traditional students were satisfied with their instructor's feedback, and 91% of students enrolled in the blended course felt the same way. Despite the reduction in regular face-to-face interaction, every blended student who participated in the focus group felt that the instructor cared for them as individuals and that the instructors were genuinely concerned with their progress in the course.

Students from both the blended and the traditional courses were asked to identify the courses' strengths and weaknesses. From the analysis of their open-ended responses, the authors found that students in the blended course liked the flexibility and convenience of the course, appreciated the quality of the multimedia course material, and felt the course improved their writing skills. One student said, “It was nice to have all the info I needed anytime I needed it. The instructor made it easier because she was always quick to answer questions and help me out when I had a problem.” Another said:

When asked to identify course weaknesses, 45% of the blended students focused on problems with technology. However, many of these technology problems occurred outside the control of the teachers since during that semester, the authors had significant problems with the Blackboard system, which functioned as the course portal. Students in the blended course also pointed to lack of interaction (15%) and poor organization (12%) as weaknesses. It is clear from our research that some students are not comfortable with a more independent learning style and depend on more frequent course meetings to pace them through the course. In contrast, the traditional students disliked the midterm (28%) and grammar assignments (23%). They also felt that the workload was too heavy (21%) for the credit they received.

We also measured the learning effectiveness of this course by collecting a portfolio of writing from students in the blended and traditional English 115 courses. The authors conducted a comparative analysis on student portfolios to determine the extent to which course type affected the quality of student writing. Throughout the winter 2002 semester, English 115 students prepared four papers:

1. personal narrative,

2. rhetorical analysis,

3. research paper, and

4. collaborative proposal.

The instructor graded these papers and placed them into a student portfolio. For the purposes of evaluating student learning, the authors focused only on the three individual papers. To evaluate student writing, the authors developed a detailed rubric to provide raters with a common reference to judge the quality of student writing.

Two generalizability studies were conducted during the Winter 2002 pilot. The first was a nested study in which three raters read each student portfolio. Sixteen readers participated in the nested study with each reader rating a total of 30 papers. The second study involved a fully crossed design. In this study, the authors had five readers read the same 16 portfolios and then compared their ratings. These sixteen portfolios were randomly selected from each of the traditional and blended sections; eight students came from the traditional sections and eight students from the blended sections. Unlike the nested study, all readers were required to read all portfolios in the sample. As a result, all five readers read a total of 48 papers.

Prior to receiving the student portfolios, all readers attended a training session. During the session, the study designers explained the scoring rubric and how to assign points to each portion. The rubric included five categories:

1. introductions and conclusions;

2. focus and organization;

3. development and detail;

4. sentence structure and word choice; and

5. adherence to convention.

Each category was assessed on a scale from 1 to 10, with 10 being the highest score possible. To retain the holistic evaluation present in English composition courses, the readers had the flexibility of assigning points within a range. The readers then received their packets with either ten or sixteen student portfolios. The readers shuffled all papers in the packet, so that all papers for all students were randomly graded and two papers from the same student would not be graded back to back. Anonymous, shuffled papers reduced the chance of the halo effect.

The preliminary results showed that 3% of the variance is caused by the paper effect. The student effect accounted for almost 50% of the total variance. Readers accounted for approximately 15% of the variance. The interpretation of the data is that readers are not consistent with each other: some readers are easy graders, while others are hard graders. This inconsistency resulted in an inconsistency in students' relative ranking, depending on the reader who read the paper. Yet, each reader was fairly consistent with him or herself. In other words, easy raters tended to be easy on all students and difficult raters tended to be hard on all students.

Overall, the papers in the blended sections had a higher average score than those in the traditional sections, with a p value of .055. In other words, at a 95% confidence interval, the scores from the blended course were higher than the traditional course. Scores from the traditional sections tended to vary the most, while scores from the blended sections were more consistent and uniform. Since the authors hoped that standardizing instruction would result in less variability in student writing quality, these results may indicate that the authors were successful in achieving this objective of the course redesign.

The authors also compared student writing according to the five categories in the rubric. In the fully crossed study, the authors found that raters graded the papers in the blended course significantly higher in (1) the quality of the introductions and conclusions (99% confidence interval) and (2) focus and organization (95% confidence interval). There were no significant differences in the other three categories: (development and detail, sentence structure and word choice, and adherence to convention). Additionally, in the nested design, the authors found no significant differences in any of the rubric categories. The nested study is less precise and is especially open to errors due to the variance related to inconsistency among the raters.

Based on the initial pilot the authors concluded that students performed as well in the blended course as in the traditional course. Indeed, some evidence indicates that overall paper quality may be higher in the blended versus the traditional version of the course.The authors are also found that there was a significantly higher rating for some important features of academic writing: introductions and conclusions, and focus and organization. Because these are global writing skills that were a focus in the blended modules, the authors feel the blended portions of the course helped improve these features of student writing.

In sum, the results of our research demonstrate that there is a delicate balance between cost effectiveness, student satisfaction, and learning effectiveness. Through this research, the authors have demonstrated that blended course redesign, including the integration of technology, can create efficiencies while maintaining the quality and effectiveness of the course. From our systematic evaluation of teaching and learning in a hybrid learning environment, the authors draw the following conclusions.

• Instructors needed more training in both how to implement the technology and how to design instructionally sound online and in-class activities

• The blended format did not damage student learning; if anything, it made their writing more consistent.

• Students have enough interaction, but the conferences are key. Blended courses work only for students who are independent learners; familiarity with technology, and writing skill are less of a factor.

• Instructors need to be more familiar with the blended course modules (1) so they use them; and (2) so the assignment objectives are clear to them, so that they can make the objectives clear to their students. New instructors should be introduced to these blended materials and instructors should be involved in the continuing development of new materials.