Facilitating Web-Based Instruction: Formative Research On Improving an Online Undergraduate Business Course

The Web offers unparalleled opportunities for student access to learning, information and communication, and thus has growing popularity as a primary medium in distance education (Crossman, 1997). Benefits include cost effectiveness from a university business perspective, convenience, and structural flexibility for the students (Jafari, 1997; Kubala, 1998). Many colleges and universities are racing to move their conventional courses online (McIsaac, 1998; Molenda & Sullivan, 2000). Despite the “virtual land rush” to grab space on the Internet and the excitement of the technology, many issues must be resolved before instruction can be delivered successfully via the Web. This study, through the process of formative research, addressed some of the prominent issues encountered by an instructor offering a course online for the first time.

There are many challenges for faculty teaching online for the first time. Using new pedagogical strategies and working with unfamiliar technology can pose unique difficulties. Often, the only feedback the instructor receives about a course is summative, which merely allows improvement the next time the course is offered. Instead of relying on data gathered at the end of the course, the researchers in this study took a more participative role by collecting data from multiple sources to inform interventions and to deal with issues as they arose.

The purpose of the study was to assist an instructor in facilitating an online course and provide timely interventions for improving the course. This study investigated what worked well, what did not work well, and what could be done to improve or solve the problems during the course and in its next iteration. The results of this study describe how formative methodology was used to address pedagogical, technological, and communication concerns in the course. The findings are relevant to instructional designers and instructors involved in Web-based distance education.

One phenomenon seen in the realm of distance education (DE) is the increased use of the Web as a primary cost-effective instructional delivery system (Crossman, 1997). When the Web was first used as a delivery system, DE was often considered to be traditional correspondence education. Instruction was (and in some cases still is) delivered through pages full of self-instructional text. Interactivity was quite limited (Hirumi & Bermudez, 1996). With the advancements of Internet technology, Webbased instruction (WBI) can now offer more flexibility in interaction and collaboration among students and instructors. The capabilities of Web-based instruction are expanding, and distance educators are starting to take advantage of the Web to incorporate sound pedagogies and viable instructional strategies to facilitate teaching and learning.

WBI, as an emerging instructional technology, requires a new approach for teaching and learning. A number of case studies have provided guidelines and heuristics on building WBI. However, an extensive literature research of DE survey models in early 1998 revealed that there were no validated models specific to DE (Williams, et al., 1999). Fitting distance learners’ needs still presents a challenge for distance educators, given the differences of needs, ages, cultural backgrounds, interests, and educational levels (Willis & Dickinson, 1997). Distance educators and researchers are exploring more theoretical frameworks to guide the planning, design, development, and delivery of Web-based instruction. In addition to the lack of a theoretical framework, implementation issues have also become very important. These issues involve the learner, the instruction, and the technology.

Many issues need to be fully investigated before WBI may become a pedagogically sound medium for delivering effective instruction (Reeves & Reeves, 1997). Issues such as information overload, becoming “lost in hyperspace,” feelings of a lack of cohesiveness, and technical problems are some of the most common difficulties for distance learners. Among institutional issues, the amount of time for an instructor to prepare and deliver the course seems to be one of the biggest concerns (Hill, 1997). Other concerns include the amount of technical support, availability of resources, and lack of training in employing sound DE techniques and methods (Shotsberger, 1997). Common issues for those who are new to online teaching are the uncertainty of instructional methods, degree of interactivity between instructor and student, and amount of guidance. While some guidelines have been established on how to facilitate online learning (Eastmond, 1995; Harasim, 1993), there is a scarcity of research studies on these implementation issues. Formative evaluation, as an ongoing assessment while the course is being delivered, can help solve these and other unforeseen problems.

Due to the nature and purpose of the study, the action research framework was adopted to guide the researchers to identify issues to be solved and implement timely interventions. Action research is designed to understand a problem, develop strategies for solving the problem, and transform the strategies into best practices. This research study involved three cycles of discovery, intervention, and evaluation (Byrant, 1996). It was a constant process of observation, reflection, and action (Stringer, 1996). Action research methodology, when used in the context of instructional technology, is also referred to as formative research, that involves an ongoing process of formative evaluation (FE) (Reigeluth & Frick, 1999). This approach was used to gather functional feedback by learning from events as they occurred and promote positive changes in a timely manner.

Formative evaluation is a process of collecting empirical data during the developmental stage for revision and improvement of instructional products (Guba & Lincoln, 1985; Weston, et al., 1995). The primary goal of FE is to improve the quality of the product being developed so that the desired goals of the product will be met. Formative evaluation involves assessment as well as judgment based on the specific information collected from the users (Beyer, 1995). Conventionally, FE is used to improve the instructional product during the developmental process, which is conducted before the instructional resources or curricula are implemented. The meaningful feedback collected during development helps shape the final product. The authors of this study, however, advocate using FE in the implementation stage of WBI because the best practices are still evolving.

In this particular study, three factors played a role in the decision to use a formative approach. First, the limited understanding of WBI instruction requires a continuous improvement approach. As mentioned earlier, there is not a validated model for WBI (Williams, et al., 1999). Distance educators are still exploring the match between instructional strategies and specific needs of unconventional students. Second, even with the best planning and development of online instruction, some unforeseen problems can occur during implementation. FE can provide a powerful tool for dealing with these unexpected surprises. Third, the instructor’s inexperience teaching online and lack of familiarity with the course delivery system made timely interventions crucial.

This research study was carried out in an undergraduate business course being offered through a mid-size midwestern university. The course was offered via Web-based courseware, Blackboard website (http://www.blackboard.com). The courseware was introduced on the campus in the fall of 1999 and was being used for the first time with this class. Fifteen students registered for this class the majority of whom were taking an online class for the first time. The professor had taught face-to-face for more than 20 years, but this was the first time he had taught the class at a distance. An instructional designer assisted with instructional strategies and technical concerns. However, there were a number of issues that needed to be addressed by using data gathered during the course.

The instructor used problem-based learning to facilitate student-centered, self-directed learning. His plan was to prompt students with real-world questions and have groups lead class discussions for the particular topics being addressed. Class activities included individual discussions and small group assignments. Most of the class communication was asynchronous. Individual activities involved participating in online discussions by answering questions proposed by the instructor. Group activities consisted of primarily mini-cases appropriate to the chapter in the textbook for that particular week. The class was randomly divided into three groups with five students in each group. Students were graded by their performance on two online exams (with multiple choice questions and essay questions), individual online participation, and group project participation.

The participants for this study included the 15 students enrolled in the course, the instructor, and the instructional designer, who was responsible for instructional and technological support.

The focus of this study was to formatively evaluate an undergraduate business course and help the instructor improve the course when it was being offered. The research questions guiding this study included:

• What issues do instructor and students encounter in teaching and taking an online course for the first time?

• How can these issues be resolved in a timely manner to improve a course as it is being offered?

• What recommendations can be offered for instructors teaching an online course for the first time?

Students’ postings were observed to determine the general trend of students’ participation. Postings included both individual replies to the class discussion questions posted by the instructor and also their discussions as a group member while collaborating with others to complete the group project.

Two surveys were designed to gather students’ feedback about the class. The first survey was administered before the middle of the semester via email. The survey included three general open-ended questions: what went well, what problems they encountered, and what suggestions they had for improvement. Nine students out of 15 students (60%) responded.

The second survey was developed based on the responses from the first survey, the interventions suggested by the researchers, and the types of interaction in distance education. Types of interaction include learner-instructor, learner-learner, learner-content, and learnertechnology (Moore, 1989; Hillman et al., 1994). This anonymous, Web-based survey was delivered by using the Assessment Tool in the course site. The survey was consisted of 19 multiple-choice questions and three open-ended questions. Twelve out of 15 students (80%) responded to the survey.

In addition to email communication with the instructor, the researchers carried out one semi-structured interview at the beginning of the project and two informal phone interviews with the instructor during the semester. The first interview was to obtain information about how the instructor would like to teach this class and with which issues he was dealing. Two informal phone interviews between the middle and the end of semester were to solicit instructor’s opinions on the collection of feedback from students and the proposed interventions based on the feedback.

Email messages sent back and forth between the instructor and the students provided another important source for understanding the issues surrounding this class. Eighty messages archived by the instructor were analyzed.

The data collected from the observations, surveys, interviews, and email messages were triangulated to determine the students’ and instructor’s perceptions of the class, issues raised throughout the semester, and recommendations for future improvement. Numerical data were tabulated by survey items and corresponding percentage in each item. The data analysis on qualitative data involved an iterative inductive process. The initial data analysis started with reading and rereading email messages and breaking the raw data into smallest sensible text units. Then these text units were coded into indexed categories and subcategories. Through iterative induction and constant comparison with categories and data, key issues and themes were generated.

The findings presented in this report reflect the process of identifying areas needed for improvement, recommendations for instructional interventions, and evaluating the course after the interventions. Observations and faculty interviews revealed issues that arose from delivering this first online course. The email survey data, Web-based survey, and students’ email communications with the instructor revealed their perception of the class and the problems they encountered. The findings are divided into three sections based on the process of formative approach: early to midsemester, mid-semester intervention, and end of semester assessment.

After nearly three weeks, there had been very few postings to the course message boards. There were only 48 posts in the first three weeks of the class, an average of less than one post per student, per week. Several of these posts were related to technical issues such as where to post the answers to the individual questions and group questions.

In addition to the limited postings, students were answering the case problems by copying out of the book. Once one person had posted an answer to the problem statement, the responses from each of the team members tended to be very similar and did not add new information to the discussion. There was little interaction occurring among the group members except when they were ready to post their answers to the class discussion board. In addition, one of the postings indicated that there was little communication occurring in other forms during the early part of the semester.

After observing the first three weeks of the class, the researchers requested an interview with the instructor to determine his perception of the class and to investigate whether other channels of communication were being used.

The initial interview was conducted four weeks into the semester and lasted two hours. After having taught classes in the field of business and insurance for over twenty years in a face-to-face format, the professor found that teaching via the Web presented a number of challenges. Underlying many of the specific issues that the professor was dealing with was a lack of time to develop and work on the class because it was being taught in addition to his normal class load.

When first considering how to re-structure the class to take advantage of the Web-based environment, the professor decided that he needed a new pedagogical approach. He researched a number of instructional strategies and decided that a problem-based learning approach would be most appropriate for the content of the course. Implementing this new approach, however, proved to be more difficult than anticipated.

The textbook traditionally used for the course was the primary resource for course material. The instructor adapted the textbook problems to the online environment and formed teams to work on solving a problem for each unit. Using this format, the professor hoped to stimulate conversation. By doing so, he believed that the students would become more engaged with the material and would have a stronger learning experience.

At the time of the first interview, there were few indications that the students were working beyond the material in the textbook to solve the problems. The professor expressed some frustration with the lack of depth to the responses and did not feel confident that this new approach was as effective as he had hoped. He did note, however, that there might have been some ambiguity in the stated objectives and expectations that the students had been given. While he had made clear that he would base a portion of the grade on class and group participation, he stated that he needed to communicate his expectations better to the class.

The technology used for teaching the class was also a factor in how well interactions were occurring. While the professor was extremely competent using many computer programs such as spreadsheets, word processors, and email, he was unfamiliar with development tools for the Web. He stated that he had done some pilot testing with the courseware in the previous semester. During the semester break, however, the university had upgraded to a new version, changing the capabilities and functionality of the program. Even though the courseware had better capabilities for supporting teams, there were some unforeseen consequences.

Not having had the opportunity to test the new system, the professor did not anticipate the students’ confusion with what message to post under which board: the Group Discussion Board (for team use) or the Class Discussion Board. He felt that these difficulties were hindering the classes’ ability to communicate with each other and with him. Additionally, he found that he did not have the time to check all of the different areas where students might be communicating thus making it difficult to track where students might be having problems.

The analysis of postings by students at an early part of the semester showed that there was very little use of the message boards, which were the intended mechanism for online discourse. When asking whether students were using any other mechanism (email, live chat, telephone, etc.) to communicate, the instructor stated that he did not have any indication that there were other methods being used.

The limited interaction was contrary to the instructor’s goal that team members would work together to solve the problems. The instructor noted that students were reading the material because their answers on the discussion boards reflected their reading. They were not, however, interacting with each other in a way that contributed to the in-depth learning that the professor was trying to achieve.

The open-ended questions that were emailed to students near the middle of the semester gave insight into their perceptions of the course. While the questions were open ended, the responses were centered on a few key issues about the class. Given the professor’s uncertainties about how well his methods and the technology were working, it was surprising to find that nearly all of the students who responded to the survey had a very favorable impression of the class.

One of the major findings in this survey was that students liked the freedom and convenience of taking a class over the Web. For some students, it was the only way they could take such a class because they had work and family commitments. Many thought the technology was being used appropriately to meet their needs and appreciated the use of message boards over email or other communication options. Additionally, the courseware was viewed as easy to use and was liked by most of the students.

While most of the respondents were satisfied with the overall structure and learning occurring in the course, there were a number of suggested improvements. Many of these revolved around the organization and expectations for class communication. Students wanted to have concrete guidelines for project work. Several mentioned that they were unsure about what was expected from their group in terms of final deliverables. There was also uncertainty about what criteria were being used to determine their individual participation in the class or within their teams. The most often mentioned concern from the students was that the professor did not seem to be giving enough feedback in the course Website. Students wanted more ongoing commentary from the instructor so that they would know whether they were meeting expectations.

Based on the major trends identified, the researchers proposed several strategies to the instructor to improve the course. While the strategies were developed as a result of the problems, they were also designed to work within the limited time budget that the professor had to make changes to the course.

The first strategy researchers recommended was to use technology to improve interaction. Improving student interaction in the course is a very complex issue and is very important in a problem-based learning class with a teambased approach. Distance students needed a clear set of guidelines for how to communicate online. The recommended guidelines included two parts. One was to tell students what technology to use and when. The second component in the guidelines was to specify how the technology was to be used. The students who were less experienced in online communication did not appear to understand what they should post in the different message areas or how to use the tools effectively.

The second strategy recommended was to provide more feedback. Given the time pressure faced by the instructor, this was a more challenging problem. Addressing the issues related to student communication might help this by encouraging the behavior the professor had hoped to occur (more student-student interaction). If, for example, students knew that there were specified areas within the course site where they could get feedback, then the professor could centralize his responses and answers to one student’s questions which might help others in the class. Such an approach would prevent the instructor from having to go into each discussion area for each group and for the class to address specific issues.

The third strategy was to provide specific guidelines for the projects. While the syllabus had clearly-stated objectives for the outcomes in the class, there was little explicit information about how evaluation would occur. In a face-to-face situation, issues like class participation can be defined on an ongoing basis, depending on the lesson for that day. Online, however, rules for participation have to be more clearly defined. More precise project guidelines would help clarify students’ uncertainty about what was expected from them in terms of group work.

The student survey was designed, in part, based on issues found in the interview with the professor and from students’ comments sent in via email. The survey was designed to assess students’ learning experience with this online course, their opinions about the online test, and other issues that concerned them. Twelve of the fifteen students (80%) in the class completed the survey. Demographic data showedthat the majority (83%) of the students had never taken an online course before this class. Interestingly, none of the students reported the courseware to be difficult to use (learner-technology interaction). The survey results helped confirm trends found in the qualitative data. The students’ opinions on group work, learning experience, online tests, and instructor’s feedback are summarized in Table 1 (interactions between learners, instructor, content).

The results on group work showed that 7 of 12 (59%) felt they were able to work effectively with their teams. When asked if working collaboratively with other team members helped them learn more than they would have worked on their own, 8 of 12 (67%) disagreed or strongly disagreed, one responded with no opinion and 3 agreed. Related to this finding was that 6 students (of 11 who responded this question) did not think that their expectations for learning the material had been met through the class. Three of them agreed the class met their expectations, and 2 responded with no opinion. Regarding the online discussions, 5 students (41%) agreed that they were beneficial to their learning experience; 2 students (17%) reported no opinion, and 5 students (42%) did not think online discussions were beneficial to their learning.

The class was somewhat divided over issues related to an online test. Two students (about 16%) of the class did not think the test content was a fair assessment of what they were supposed to learn, 5 students (42%) responded “no opinion” about this issue, and 5 students (42%) thought the test was fair. As for appropriateness of test format for this online course (timed multiple choice questions and essay tests), 4 students (33%) responded “disagree,” “no opinion,” and “agree respectively.”

Students’ opinions on instructor’s feedback were also divided. Five students indicted that the instructor provided sufficient feedback, 3 students responded no opinion, and 4 students did not think the feedback was sufficient.

Students’ responses to the opened-ended questions revealed more detailed information about their perception of the course. These and the analysis of results from students and instructors’ email messages are summarized below. These qualitative data gathered from students’ survey informed the quantitative results of students’ perception of the class.

One of the consistent findings is that students enjoyed the flexibility and convenience of taking an online course. Other findings include that students also liked the professor’s patience and understanding in dealing with problems they encountered. Some students enjoyed the group work and online discussions. Three prominent themes regarding problems and concerns emerged from the qualitative data analysis.

Technical concerns. Although the courseware was relatively user-friendly, students constantly ran into technical problems throughout the semester. Several students had difficulties in taking the online exam. The restriction of the exam format and lack of flexibility in test sequence caused some students not to perform as well as they expected. Some students got a zero on particular sections due to technical restrictions. The instructor had to reset the test for these students. Other technical problems included lost connections during the test or transferring files, and posting and locating messages at the right place.

Students reported in the survey that they did not think Blackboard was difficult to use. The email messages sent back and forth between students and instructor revealed that there were more technical problems than questions about assignments or reading. The instructor had to spend a great deal of time resolving these technical problems.

Assessment. Students were very concerned about their grade, and how they were evaluated on the individual participation and group participation. Several students emailed the instructor asking what grade they might get and how they could improve their grade. In addition, students were concerned and some were frustrated by the format and technical difficulties found in the online test. More than a couple of students mentioned that they were not good test takers even if they knew the material well. The timed exam made it more difficult.

Group work. This class used small groups to work on mini-projects. It is not surprising that there were some complaints about uneven participation. “Free riders” (group members who made no contribution to the group task but got the credit) were rather vexing for the students who contributed more for the group project. Some students were concerned about this fairness of the workload in the group assessment.

A couple of other issues, not reflected by majority of the students, are worth mentioning. Two students thought that the online class lacked social interaction. They felt they would learn more through hands-on and face-to-face classes. In addition, insurance is a hard subject (a comment made by one student), and not being able to discuss it in class (face-to-face) made it harder to understand.

Through our informal conversations and semi-structured interview, the instructor expressed that formative evaluation was a very good reinforcement. He stated that it was good to have an independent source to provide feedback. Students seemed to like this idea and were quite open in talking about their issues and concerns. He thought a mid-term formative evaluation and recommendations were useful. If he had time to implement them, he projected these recommendations would have helped improve this class. After the class was over, the instructor had mixed feelings about his first experience teaching online.

The instructor did not think that students accomplished what he wanted. He expected them to go beyond the textbook, but only one or two students did that (consistent with his comment earlier in the semester). He also felt that students did not learn as much as they would have in an on-campus class. With his on-campus face-to-face class, he was able to present extra material. With this Internet class, although databases were provided, very few students used them. But one of the things that instructor really enjoyed was the rapport he and some of his students established through emailing back and forth to address questions and concerns. He found this online rapport was quite rewarding.

The researchers began the study by observing the visible interaction occurring on the course Web site. Analyzing the evaluation data identified three general major difficulties. First, very limited interaction occurred on the course message boards during early part of the semester. One of the reasons for this apparent lack of student involvement was that most of the students had never taken an online class before, and they had no experience in how they should communicate within the courseware. Additionally, although the instructor was aware of the lack of student involvement in the course, his limited time and heavy workload kept him from putting more effort into facilitating the students’ online communication.

The second major problem was the amount of feedback students were getting from the instructor. The same factor (time shortage) also affected the amount of individual feedback the instructor was able to give to the students. The professor hoped that students would interact more with each other and would learn from these interactions, rather than relying on him for the majority of the feedback.

The third major issue that emerged was related to standards for the class. Students indicated that they were not sure how they were being evaluated. This uncertainty was especially true for the participation portion of their grade. Clear and specific guidelines about how participation was being evaluated (for example, by number of message board postings, depth of the posts, etc.) would help clarify the students’ uncertainties. Likewise, providing explicit guidelines for group projects would have helped address students’ concerns about group work, which could lead to more participation from team members.

There was a high degree of congruence between the instructor’s beliefs and student attitudes on these three major issues. The instructor, however, had a much more negative view of how the class was progressing than the students did. He felt that the students were going to be very disappointed with the overall experience they were getting from the class. In contrast, email messages to the research team and other survey data revealed that while students had specific issues they wanted to see improved, they were generally positive about the class.

In terms of soliciting students’ feedback about the class, both the researchers and instructor sensed that students seemed to like the third party as independent source to evaluate the class. This reflected from students’ lengthy messages to the researchers, their comments and suggestions about the class and their expressions of willingness to help. Students also felt appreciated that they were asked for feedback before the middle of the semester. The instructor felt that the third party evaluation was more objective and allowed him to see the class from a different angle. The instructor noted that some of his feelings about the class were verified by our findings, while others were refuted.

This course presented a couple of challenges for the instructor. The first challenge was time management. Due to his busy schedule and heavy workload, he was not able to facilitate the class discussions and group activities as effectively as he wanted to. Internet classes demand more time from the instructor to communicate with students. In addition, he had to spend much time on answering students’ technical questions. If he had been freed from dealing with these technical responsibilities, he would have been able to put more effort into the development of the course. Another challenge the instructor faced was finding the best fit of instructional strategy to this particular introductory course. There is very little literature about using PBL in Web-based instruction. Knowing how to implement PBL in an Internet class and how to facilitate group problem solving presented a great challenge for the instructor. With limited guidance from the literature, the instructor had to explore the strategies through trial and error.

Given the context of this research study, the instructor’s inexperience with the course delivery medium, teaching online for the first time, and lack of time and support, formative evaluation appeared to be essential in dealing with the issues that arose as the class was being offered. The results of formative evaluation not only helped instructor to have a better sense of how the students were doing, but also provided the basis for timely interventions to improve the course. Although the instructor felt that the recommendations for instructional interventions were very good, the limited time (teaching on top of normal workload) and support he had prevented him from fully implementing all the interventions. The researchers felt that institutional support (such as reasonable workload for online instructors, sufficient technical support, etc.) is a key factor for successful implementation of any interventions.

Through this formative study, some recommendations can be made to the instructors teaching online course for the first time. First, instructors need to be able to or be willing to spend more time on the Internet course. Online course demands more time from the instructor for giving feedback, facilitating class activities, and providing instructions on learning materials. Second, the instructors should anticipate some unexpected problems and issues. In the case of this first online business class, some problems were the result of students who were inexperienced with the technology. The instructor was flexible in dealing with these problems, which reduced students’ anxiety over technical concerns. Third, sufficient technical and instructional support plays an important role in ensuring the success of an online course. Online courses rely heavily on technology for communication and collaboration. Instructors may not have enough time to learn all the capabilities of course delivery technology or to deal with technical questions. The new learning environment also requires the instructor to use different strategies to facilitate learning. For many instructors new to teaching online, instructional support is in great demand.

The instructor in this study was very motivated to try innovative teaching techniques and strategies. He spent time reading pedagogical literature and pilot testing the technology before teaching the class. Additionally, the class was being taught voluntarily, despite his already busy schedule. Even with a highly motivated faculty member, however, implementing an online class is a difficult task that requires new skills and strategies for success.

In a face-to-face class, knowing whether students are learning is made easier because the instructor can often read the facial expressions, react to comments, and then make changes on the fly. A course being taught online, however, does not allow for quick adjustments and there is little opportunity for the instructor to make intuitive judgments about how well the class is going. By implementing some form of evaluation during the class, the instructor can address problems students might be facing in a timely manner.

Many Web-based courses being built today have little in the way of automated feedback systems. Even the sophisticated course delivery systems that are not available do not adequately track students in a manner that allows the instructor to make meaningful improvements to a course while it is in progress. Adding a means for students to anonymously give feedback during the semester can be a powerful tool for the instructor. For those working on limited budgets and without sophisticated electronic tools, an objective third party such as a teaching assistant or instructional designer can solicit the feedback from students through email. A simple analysis of data can yield important information about how well students are progressing through a class.

As universities continue the mad dash to offer their courses online, reasonable timelines for design and development should be taken into account. If faculty have to use untested methods on constantly shifting courseware with their students, sound instructional practice would suggest that this is not good for the students, faculty, or, in the long run, the university. The reality is, however, that there will continue to be uncertainty about the best practices for online course delivery for the near future. Formatively evaluating their courses and making improvements throughout the semester can reduce some of the problems caused by the rapid changes that are occurring in distance education.