Decisions, Decisions: Choosing a Web Course Development Package

Since 1998, faculty members at the University of Kansas have had free access, through a university server, to an online course development package. During this time, the capabilities of online technologies have multiplied, the computer literacy of the average faculty member has risen, and the use of online teaching resources has demonstrated a commensurate increase. One of the outcomes of these rapid developments was that the software originally purchased for the creation of online courses and supplemental materials appeared to no longer meet the needs of a professoriate exhibiting a wide range of technology skills (and an even wider range of motivational levels) for designing and creating Web-based instruction.

Although the template-based package worked exceedingly well for many faculty members who appreciated its easy-to-use format, others chafed at structures they perceived as constraining and inflexible. In short, the more technologically adept instructors soon found themselves bumping their heads on the software ceiling as they attempted to customize their online offerings. In addition, as news of technical glitches (disappearing data, for example) circulated among the faculty, fewer and fewer instructors were willing to put in the time to create Web-based materials that students might never find, and technical support staff were increasingly frustrated in their attempts to keep technical problems at a minimum. Therefore, a review of several integrated course development packages was undertaken (including a newer version of the existing product), with the hope that a set of teaching tools appealing to technology novices and prodigies alike could be identified and acquired.

The intent was to consider, initially, any Web development package or any set of development tools that could be combined to meet the instructional needs identified by faculty. Ultimately, only integrated development products—those containing all of the desired instructional and communications tools in one package—were reviewed, based on long-term goals for administrative integration and navigational consistency.

The major stakeholders in this process included the university faculty, technical support staff in Academic Computing Services (ACS), and instructional support staff from Instructional Development and Support (IDS). IDS is a relatively new university office, created in 1998 by merging the traditional Media Services center with a faculty computing support facility and including an instructional design component. Since that time, the focus of IDS gradually has shifted from a hardware provider basis to an identity that reflects an emphasis on learning environments, instructional planning, and technology integration. In this situation, the director of IDS was expected to make the final recommendation for the purchase of an appropriate development package, oversee faculty support (e.g., training and consultation), and disseminate information to faculty about using online course materials.

Academic Computing Services provides Internet access and e-mail services to the university, equips and oversees on-campus computer labs, offers training and software documentation, and staffs the computing help desk. Once selected, ACS would be responsible for installation and maintenance of the course development software, and would share responsibility for training. Technical support staff from ACS who had been assigned to troubleshoot the previous Web development package were eager to evaluate the software candidates for evidence of their reliability, consistency, and technical soundness.

Faculty were offered the opportunity to participate in the software review process and the response was encouraging. Disciplines represented included communications studies, electrical engineering and computer science, foreign languages, biology, English, economics, music, and special education. Several instructors who had been using the current development product volunteered, along with those who had used other software tools (FrontPage or Dreamweaver, for example) for creating online course materials and building Websites. Two of the volunteers had taught fully-online courses and the remaining had posted static information (e.g., syllabi, notes, or graphics) and/or had utilized interactive features such as discussion groups and online quizzes.

Involving representatives from a variety of academic disciplines and faculty support services required high levels of organization and flexibility, but was, ultimately, the most important element of the process. Michael Henry, Associate Director of Project Development at KU’s Edwards Campus in the metropolitan Kansas City area, remarked, “It was exciting to see this kind of collaboration, focused around finding appropriate Web course technology for our specific needs. It was a good exercise for us, even if we decided not to select a Web course server.” The strength of this review process rested primarily in the perspectives of those who were eager to contribute their ideas and support.

Rather than reinventing the wheel of evaluative criteria to use for this project, existing matrices and side-by-side comparison studies done at other institutions were reviewed (see, e.g., Hazari, 1998; Marshall University, 2000; University of the Future, 2000; Wichita State University, 2000). From these, an initial set of preliminary criteria was derived to which the reviewers could react and prioritize according to their needs, biases, and interests. For example, Dee Vernberg in Communications Studies confessed, “I especially like the quiz features of these software packages. Two days ago I got an e-mail from a student [that said], ‘I just wanted to say that the online quiz was a good idea. It's already hard enough that this class is very fast but for you to give students learning tools like the quiz is awesome. Thanks.’” This perspective, and others, helped shape the early attempts at creating a finite set of review criteria.

From these beginnings, two sets of measures evolved. The first was considered baseline criteria (see Figure 1) and included basic features that needed to be present in any product selected for further consideration. These were considered the commonalities among the packages that were reviewed and included things like threaded discussion capabilities, online help for students, and password protection for course materials.

The second, more comprehensive, list was broken into “tools and features” (see Figure 2) and “general characteristics” (see Figure 3), each of which had several sub-topics included. This series of items would be refined as the process evolved and serve as a gauge, or standard. The Tools and Features array included Instructional, Collaborative, Authoring, and Course Management sub-topics, focusing on the specific capabilities of the package within each category. For example, within the Collaborative topic, criteria related to threaded discussion, e-mail, and chat were listed. The organizational structure proved to be an excellent device for collecting and sorting the most important of the criteria, even though many of the items could have fit into more than one category.

The General Characteristics section included several factors that were subjective in nature (e.g., attractive interface) and the sub-topics—Usability, Technical Factors, Customer Base, and Costs—were meant to establish a big-picture perspective. This list was helpful in teasing out the details of what each category meant to the group involved in this process (e.g., “Usability” included the more specific factors of “customizable passwords” and “text-only version available”). It is important to note that each section of the comprehensive list of review criteria gradually evolved into its present form after input from faculty and support staff about the relative importance of individual items. Comparison tables and extensive matrices prepared by others were often much more detailed, but the process of winnowing the criteria proved to be a useful exercise and ensured that the project remained manageable.

In addition to meetings of the “official” software review team, input regarding the selection criteria was gathered using other means, as well. A survey of faculty who had attended a workshop series on using technology in their teaching produced useful data related to the value of specific online tools (discussion groups, synchronous chat, etc.) and helpful comments regarding training formats and schedules. Much more detailed feedback came from a series of open meetings attended by faculty from a variety of disciplines to discuss their needs, suggestions, and concerns about online courses and tools. Finally, approximately a dozen faculty members, who were unable to attend one of the open meetings, called or e-mailed IDS with their input.

Identifying software products to evaluate for possible adoption was probably the least difficult step of the entire process. Online education has become so popular that articles, Websites, discussion lists, conferences, advertisements, and plain old word-of-mouth make it almost impossible to avoid hearing about any number of course development packages. Five products were identified, initially, that met the baseline standards, although others were later discovered (and eventually discarded as options). Of the initial five, two were eliminated because they were too expensive; when one of the remaining three software companies represented acquired one of the others, the pool narrowed to two candidates. It was these two packages that were most rigorously evaluated.

Faculty and support staff representatives were asked to visit the Websites devoted to each software package under consideration to familiarize themselves with the overall look-and-feel of the products. The reviewing team did this in pairs, using the previously determined criteria to guide their progress through the materials. Faculty representatives also attempted to use the “free” versions of each product to begin building courses. This was followed by a group discussion during which various perspectives on the packages were shared and perceived strengths and/or weaknesses identified. Unfortunately, the team eventually discovered that the online version of each software package was not the most recent available. This may seem like a minor point, but the initial impressions formed by the reviewers colored their viewpoints for the balance of the process. Subsequent evaluative activities, therefore, were undertaken with assumptions based on obsolete information that influenced team members’ opinions regarding each product.

At this juncture, several philosophical issues surfaced related to the levels of institutional support provided to faculty for developing innovative instructional materials. One of these issues, in particular, proved helpful in clarifying the focus of the group and its long-term goals. This was the idea that the easiest-to-learn package might not be the best choice, in the end. While this may seem obvious, at first glance, it generated a discussion of how much an instructor can reasonably be expected to learn as a part of his/her duties. Few jobs these days require no new skills, retraining, or retooling in order to remain productive, but how much could be considered excessive? Is it realistic to expect every instructor to learn HTML? How about word processing? E-mail?

This fruitful debate originated from a suggestion that a way to evaluate the products might be to have novices attempt to perform certain “course building” tasks (e.g., upload a document) while being timed. After briefly considering a modified version of this activity--minus the timer--it was concluded that by doing this, the emphasis would fall on the simplicity of the product rather than the robustness of its features, an outcome that could be counterproductive in the long run. Not surprisingly, the group never resolved the broader issue of the faculty’s responsibility for learning new teaching strategies and technology skills, but the discussion ultimately tempered the selection process.

Although the software products that were considered are intended for use with individuals at a distance, a final selection decision would be made only after an in-person, on-site demonstration by each company’s representative(s). This may seem paradoxical, but is, in fact, a natural consequence of the innovation decision process (Rogers, 1995). As the decision time for adoption of a product draws closer, the influence of interpersonal communication channels grows, and that of mass media weakens. Thus, the personal touch of an on-campus visit is a near-mandatory part of the process. In this particular instance, it would be especially important, because there was no clearly superior product under review; both were seen as excellent packages and viable contenders.

The two companies that arranged site visits to show off their products exhibited strikingly different presentation styles and strategies. The first was extremely low-key, with a lone presenter demonstrating the product in an informal manner and answering questions as they arose. During the presentation, time was set aside for a telephone conference with a software engineer at the development headquarters to answer technical questions and discuss upcoming improvements.

The other package was shown by a team of four individuals who all participated, to some extent, in the presentation. This group had a faculty member from an institution of higher education demonstrate the software, utilizing another of the tenets from innovation adoption theory: persuasion is more likely to be successful when we perceive the change agent to be like ourselves (Rogers, 1995). Extensive print materials describing features, training options, technical specifications, and the like, were distributed. Tee shirts and coffee mugs bearing the company’s logo were also passed out to those attending.

Questions asked of the presenters in each session reflected the priorities that had been identified among the criterion measures during earlier discussion and review. Simplifying the tiresome exercise of creating, uploading, and manipulating mathematical and statistical information was a recurrent theme, as was the need for foreign language character sets, customizable displays for content presentation, flexibility in assessment options, and easy integration with PeopleSoft, the administrative software used at KU.

Unfortunately for the review team, the on-site visits did little to distinguish which product would be the superior choice--both appeared to be soundly designed and well-supported. There were minor differences in feature sets, but these weren’t significant enough to draw a clear preference between the two. The decision would rest on a balance of factors related to the software quality and what might be termed “external factors”: those that looked beyond the software itself to institutional priorities and long-term plans.

One final meeting of the review committee was held to discuss the two products, revisit the evaluative criteria, and to hear the more subjective reactions to each package that might not otherwise emerge. Although there were individuals who favored one package or the other, none of the committee members felt strongly enough to attempt to persuade the others of their opinion and consensus was reached relatively quickly. The product recommended by the group was selected eventually for institutional adoption.

The reasons for selecting the package chosen were:

• Easy-to-learn features and an attractive, streamlined user interface;

• Compatibility with existing online courseware and ease of migration from one format to the other;

• Potential for extensive integration with the institution’s administrative software, including an aggregated interface for academic and administrative services; and

• Established customer base in Kansas (potential for collaborative activity).

Although this was merely the beginning of our efforts to facilitate the increased use of online courses and course materials, several suggestions can be drawn from the experience. First, involve as many stakeholders as possible--especially faculty members. Balance the group by including those who are comfortable with high-tech applications and those who’ve not yet made the leap to online instruction. Also, include those crucial staff members who will provide technical support. Their input could mean the difference between a relatively smooth implementation and one filled with delays, glitches, and needless stress. At the University of Kansas, an institution with a history of decentralized decision making, soliciting input from a variety of sources not only provided a stronger rationale for the final recommendation, but also enriched and strengthened the links among the groups represented.

The second suggestion is to prepare a comprehensive list of possible evaluative criteria, then prioritize them according to the needs of the users, the goals of the administration, and the long-term plans of the institution, weeding out the less important points. This will help to contain the process and focus the efforts of those involved. An associated hint would be to not rely solely on objective, checklist types of data. Qualitative, subjective responses to the product under review can also provide useful guidance to the decision makers.

Finally, keep in mind why course development software is being considered for adoption. Is the institution planning to launch a series of online degrees or is it more interested in providing a greater range of instructional options for traditional courses? Is there strong interest in and support for high-tech applications (streaming video, for example)? Are administrators hoping to increase enrollments while cutting costs? Will the institution’s network infrastructure sustain a dramatic increase in high bandwidth traffic? These and other questions can guide the selection process and influence the assumptions under which the reviewing team operates. At KU, the primary focus of the software evaluation project remained on increasing and enhancing instructional technology options, whether for online courses or supplemental materials in traditional, face-to-face classes. Therefore, any software product considered by the team had to include a wide range of features to support multiple instructional strategies and styles.

Now, the real fun begins, as the installation, training, and instructional development components of the process take over. Time and experience will determine if the final selection decision was sound--but the value of the software review, in terms of strengthened ties among the participants, remains to provide a basis for future collaborations.