AN INSTRUCTIONAL DESIGN APPROACH FOR EFFECTIVE SHOVELWARE: Modifying Materials for Distance Education

E-learning solutions are rapidly becoming an integral part of many university courses, continuing education programs, and training programs in business, government, and the military. A number of experts predict that, in the training area, the number of courses delivered by technology will increase. Cone and Robinson (2001) suggested that the number of e-learning courses was expected to double by the year 2003. Rosenberg (2000) suggested that the expenditures for courses and programs delivered using technology have increased 66% during the previous 5 years. In the private sector a number of companies are reporting the economic benefits of e-learning. As a result of e-learning solutions, IBM saved $20 million in 1999, while Ernst and Young reduced the cost of training by 35% (Strother, 2002). Strother also makes a strong case that corporations are increasing their emphasis on e-learning solutions.

We believe that is necessary to assure that e-learning trainees and students receive courses and training experiences that are well-designed from an instructional standpoint. Cone and Robinson (2001) have identified two problems with e-learning, including “poorly designed e-learning” and “insufficient focus” (p. 2).

In this article we will focus on the problem of poor design or lack of instructional design in e-learning materials. In many cases: (a) training courses, college courses, and other traditional instructional packages are repurposed for use on the Web, or (b) faculty/trainers develop a new course for Web delivery that does not provide adequate instructional support. What really happens in practice? We believe that, for both repurposed courses and newly-developed courses, frequently information is collected and assembled in the form of online course syllabi, class schedules, course notes, assignments or projects, PowerPoint slides, or course module. This information is then posted to the Web using software such as WebCT, Blackboard, Lotus Learning Management System, or other software designed for the purpose of delivering e-learning instruction. In addition, threaded discussions, chats, and e-mail are also incorporated as a strategy. We believe that “courses” delivered in this manner are not necessarily instructional, and that they should really be labeled “Web-based information.” Little or no instructional design may have been done ensuring that sound instructional strategies are embedded in the instructional materials. Such materials are labeled as shovelware. Shovelware is defined as “content taken from any source and put on the Web as fast as possible with little regard for appearance and usability” (whatis.techtarget.com). For example, an instructor can collect information and shovel it into an application such as Blackboard or a learning management system to create a “course.” The key to high-quality instruction rests with effective instructional strategies developed in the context of a sound instructional design model. Good strategies are typically missing from shovelware courses.

The primary purpose of this article is to provide a tool that will help instructional designers determine if existing e-learning courses and sharable content objects are well-designed instructionally, or primarily a collection of information posted as a “course” (shovelware). (We will use the term “course” to refer to modules, sharable content objects, and other forms of instruction.) In addition, we will discuss how the tool can be used to evaluate the instructional soundness of a traditional course that is being repurposed for Web delivery. We label the tool the instructional disassembler. We believe that the instructional disassembler tool will help developers ensure that existing e-learning courses are instructionally sound and not just collections of information presented in an e-learning environment. The instructional disassembler tool will also help designers repurpose traditional courses for elearning environments. In particular, instructional designers can use the instructional disassembler tool to identify the component parts of a course in order to describe the content included and if strategies are incorporated in an existing e-learning course or an existing traditional course. Specifically, the tool will allow the instructional designer to disassemble and examine the content, information, and instructional components included in the course. The designer can then determine if additional or initial instructional design work is required. We will first emphasize the importance of defining the construct of distance education and briefly discuss issues concerning distance education in higher education, business, and the military.

Distance education has several different definitions that can include either synchronous and/or asynchronous communications. We are using Keegan’s (1996) definition in which he makes a distinction between distance education and virtual education. Keegan’s definition of distance education separates the learner and instructor in time and location on a quasi-permanent basis. Keegan allows for two-way communication and occasional meetings, but places a greater emphasis on asynchronous communications. When synchronous communications, such as two-way video and audio or satellite delivery systems are the dominant mode of delivery, Keegan labels the delivery as a virtual system rather than a form of distance education. Our focus in this paper is on asynchronous distance education courses that require the development of materials prior to the start of the course offering and allow for anytime/anywhere learning.

Universities are striving to increase their online offerings to remain competitive in a market that is no longer defined by geographic boundaries between universities. The competition and search for students goes beyond trying to fend off other universities who establish a regional center in your own “backyard” to attract students. Today, a student can take a course from a university located almost anyplace in the world without leaving home. In recent years, we have seen many universities increase their online offerings or create a separate entity for delivering distance education courses. In 1997-1998, there were approximately 50,000 college-level credit courses offered in the United States via distance education (National Center for Educational Statistics, 1999). In 2000-2001, there were approximately 3.1 million enrollments. The number of postsecondary schools offering distance education courses has doubled since 1995 (National Center for Educational Statistics, 2003).

One strategy seen in university courses is the adaptation of existing classroom courses for Web-based delivery. Central to most university-level courses are textbooks that form the basis for many courses. The lectures, exercises, and readings are often wrapped around the textbook to provide a course of instruction. Individualization models such as Keller’s Personalized System of Instruction (PSI) often used the textbook as the primary source of information. Similarly, the University of MidAmerica and Coast Community College often “wrapped” a distance education course around an existing television program. For example, their introductory psychology course was based on a series of Psychology Today films, the poetry course Anyone for Tennyson was based on Nebraska Educational Television series by the same title, and the American Nationalism radio course was based on series of lectures by Henry Steele Commager. In the last two examples, textbooks were developed to support the television and radio content. Given these examples, it is clear that instructional designers in higher education have repurposed traditional courses and information for distance education delivery for a number of years.

In business and the military, there is a trend to move instruction from the classroom to the Web by converting the classroom materials for an e-learning environment. One reason for this move to convert traditional courses to Webbased courses is to increase effectiveness (Cone & Robinson, 2001) and to reduce costs (Strother, 2002). Like higher education, businesses often convert traditional courses or simply information into Web-based courses. There are several references on the Internet that describe the conversion of traditional classrooms to e-learning courses in the private sector (e.g., http://www.learningcircuits.org/may2000_elearn.html). Similarly, there is a move in the military to use e-learning as the primary means of course delivery. The U.S. Army lists over 1,500 e-learning courses for training (http://usarmy.skillport.com/).

Course conversions from the traditional classroom to a Web-based format can take different approaches. For example, rather than providing a lecture, an instructor might provide lecture notes or a PowerPoint presentation on a Website. Another instructor might provide a streaming video of the lectures. Similarly, classroom exercises might be converted to readings rather than attempting to provide an interactive exercise online.

As a result, Web-based courses may include a considerable amount of information. The information is accurate and includes much if not most of the information needed by the student. This information, however, is not always “instructional.” It may not be structured for optimal learning. The instructor and students typically provided the instruction in these courses as they interacted in the classroom. For example in Keller’s Personalized System of Instruction (PSI) course, the instructor and tutors provided the additional instruction to help the learner grasp and understand the content. Simply placing the textbook in the hands of the student and the tests online with automatic scoring would neglect the instructional component of the classroom-based course. In some cases, the classroom course was not well designed in the first place. Cone and Robinson (2001) suggest that the failure of many e-learning courses to meet expectations is due to the poor design of the course.

Many e-learning courses are composed simply of information that is stored online using course management software. Is there a way to determine if e-learning courses are instructionally sound or if they are simply shovelware? Is there a way to use an instructional design model to design effective e-learning instruction rather than producing shovelware? How can we as designers improve the effectiveness of instruction that is based on existing print materials without incurring significant costs? In this article, we will describe an instructional design tool that an instructional designer can use to evaluate the instructional components of existing elearning materials, and convert existing traditional courses and instructional materials for Web-based instruction.

We believe that traditional instructional design approaches and models have much to offer designers of e-learning courses. One approach is to modify the traditional ID model, which starts with front-end analysis. In a traditional routine use of an instructional design model, initially, a front end analysis is completed and the designer and subject-matter expert work together to define the instructional goals and to define the content needed for the course. The most important aspect of this analysis is the task analysis step, which may seem superfluous with content that is already defined by an existing course. We are proposing a modification of the traditional instructional design model to increase the functionality when evaluating existing e-learning courses and when repurposing traditional courses for Web delivery.

Although the existing course materials already include the content for repurposing to a Web-based course, we cannot neglect the task analysis step. Task analysis helps the instructional designer identify the structure of the content for both traditional and e-learning courses so that appropriate strategies can be developed. However, the traditional task analysis process of working with the subject-matter expert may not be appropriate when repurposing traditional course materials or existing Web-based instruction for e-learning. A process is needed that will help designers determine the content structure of the materials and if the existing content is adequate.

How do we determine if the content of existing traditional courses and print materials is instructionally sound, including the critical information and strategies needed for learning? Reverse engineering is a term used to describe the analysis of an existing product (i.e., a toaster oven) or software to determine the component parts and the function of each. As instructional designers, we suggest using a reverse engineering process to determine the structure of existing e-learning courses as well as materials that will be repurposed for elearning delivery. “Reverse engineering is the process of analyzing a subject system to (i) identify the system’s components and their interrelationships and (ii) create representations of the system in another form or a higher level of abstraction” (“Bibliographic references: Terminology,” 1997). A subpart of the reverse engineering process is design recovery that adds domain knowledge and fuzzy logic to the reverse engineering analysis to identify higher abstractions (“Bibliographic references: Terminology,” 1997).

Programmers use disassemblers to reconstruct the commands, flow, and logic of compiled programs. This disassembly process is referred to as reverse engineering. A similar process of reverse engineering can be applied to existing instructional materials to identify the structure, information included, and the sequencing of the instruction. This process is a form of task analysis that operates on existing materials rather than the traditional approach in which a subject-matter expert is interviewed.

Jonassen, Tessmer, and Hannum (1999) identified three task analysis methods that have potential for use as reverse engineering tools. These three methods are described in the following paragraphs.

Decompose, Network, and Assess (DNA). The DNA approach is a cognitive task analysis process for eliciting knowledge and skills from experts (Jonassen et al., 1999). The DNA methodology requires a software program that interacts with an expert to decompose the domain knowledge. The elements identified through the decomposition process are then ordered and linked in the networking module. Finally, other experts validate the analysis in the assess module.

DNA is not practical for reverse engineering the existing content, as it depends on interaction with an expert who actively answers questions. Attempting to do the task manually on printed materials would appear to be an extended process with unpredictable results.

Matrix Analysis. This methodology was first introduced as a task analysis method for developing programmed instruction (Evans, Homme, & Glasser, 1962). Thomas, Davies, Openshaw, and Bird (1963) further developed matrix analysis as a reverse engineering process with procedures for analyzing programmed instruction. This process was further refined by Davies (1973). Matrix analysis involves four general steps (see Figure 1). First, a rule set is constructed by analyzing each frame of the programmed instruction. Second, the rules are numbered and the numbers are recorded on a diagonal or both axis of a large piece of graph paper (an alternative approach is to use an application such as Excel and shade the cells). Third, each rule is compared to each of the other rules. If a relationship exists between the two rules, then the two cells representing the intersection of these two rules are shaded. Fourth, the matrix is then analyzed to identify content structures such as chains, discriminations, concepts, and principles based on the visual pattern created by the shading of the cells.

Although matrix analysis was developed for use with programmed instruction, it may have potential with other types of instructional materials. However, its application may be more difficult due to the lack of the frame structure prevalent in programmed instruction. A drawback to using matrix analysis is the tedious and complex nature of the matrix for complex material.

Syntactic Analysis. Syntactic analysis is one of the few methods developed to analyze existing materials. Syntactic analysis was developed for analyzing reading and linguistic information. While there are no examples of its application for task analysis (Jonassen et al., 1999), it may have potential as a task analysis method. This method is used to identify tasks and objects (i.e., subject, verb, and modifiers related to the task). The analysis can produce a database of the task units and the related objects. One can then manipulate the database to identify relationships and structures between the subjects, verbs, and modifiers of the various tasks.

Syntactic analysis may reduce the information to too fine of a level for the purpose of task analysis. The value of specifying the subject, verb, and modifiers of each task statement is not readily clear for the purpose of task analysis.

DNA, matrix analysis, and syntactic analysis all appear to be adaptable to a reverse engineering approach. However, we feel that each of these approaches adds an additional level of complexity. This complexity can translate into additional analysis time and resources that may not be available to the instructional designer or design team working on a shovelware project. We are proposing a simpler approach that we believe can be as effective for the design of Web-based instruction. This approach, the Instructional Disassembler, is used to reverse engineer instructional content or information.

Designers can use the instructional disassembler to determine the content and strategies used in an existing e-learning course or traditional course that will be repurposed as an elearning course. For example, “shovelwared” courses may or may not include instructional strategies that facilitate learning. Similarly, the content may not be adequate for teaching a concept or principle. The instructional disassembler is a tool for reverse engineering the course into smaller units for identification of the content and strategies. Once disassembled, a designer can determine the effort needed to improve the quality of the course.

Instructional Disassembler. The instructional disassembler is used to break down the existing content in existing e-learning courses, traditional courses, and materials including textbooks or study guides used in a course into component parts. Designers can then analyze the content to determine if adequate information is provided. For example, Tennyson and Cocchiarella (1986) suggest that to teach a concept one needs to include the label (i.e., category name), definition, and one best example. By examining the disassembled content, the designer can determine if these three elements are included for each concept. If they are missing, modifications or additional information can be added, such as a supplemental material.

The effectiveness of this method is highly dependent on the instructional designer. It is assumed that the designer has the expertise to identify content structures such as facts, concepts, principles, rules, and procedures. The designer must also be familiar with strategies one might use to facilitate student learning for each structure. If two or more designers disassemble the same content, we would expect them to negotiate differences in their interpretations.

Disassembling content, information, and instruction involves three steps. First is the disassembly of the content. Second is identifying content structures. Third is analyzing the instructional adequacy of the content. The following sections detail each of these steps.

Disassembly of Content. Disassembling content, information, and instruction are all done using the same approach. This step of the process is similar to the traditional task analysis used when designing instruction. However, disassembly is done on existing e-learning or traditional course materials rather than starting with a blank slate and querying a subjectmatter expert. The process starts by breaking the content into the smallest units possible (we are using the term content to include instruction, content, and information). These units or phrases are then recorded in a traditional outline format. Major sections such as first-level headings are assigned to the highest level (i.e., I, II, III) of the outline. Information within the section is then placed in the sublevels of the outline (i.e., A, B, C, 1, 2, 3).

The following are heuristics for disassembling the information.

1. Break each idea into its simplest form. Disassembling should result in phrases or single idea units rather than complex sentences. That is, a single sentence might be reverse engineered into several smaller idea units.

2. The outline produced by the disassembling does not need to reflect the same sequence as the information. Idea units may be reordered to reflect “raw” content that is organized around knowledge structures rather than instructional strategies or elaborated writing.

Consider the following example concerning the concept “loan discount” that includes the disassembly of the content information. First, the “raw” content is presented. Second, the disassembled content for “loan discount” is listed.

Notice how the first sentence, “Also often called ‘points’ or “discount points,’ a loan discount is a one-time charge imposed by the lender or broker to lower the rate at which the lender or broker would otherwise offer the loan to you,” was disassembled into three idea units.

Another example of content disassembly is presented in Tables 1 and 2. In Table 1, selected content information included in a document describing how to buy a home is presented (U.S. Department of Housing and Urban Development, 1997). The content for the mortgage document is presented in Table 2.

An examination of element B, loan origination, illustrates the guidelines presented. First, the complex sentences are disassembled into individual idea units. Second, the disassembling organizes the information in a logical manner rather than the sequence used to present the content. By following a logical sequencing, we can group related idea units that will make it easier to identify the content structures.

The two remaining steps to complete are “Identifying Content Structures” and “Determining the Adequacy of Instruction.”

Once the content is dissembled, the designer can identify the content structures in the instruction. The Instructional Disassembler uses the content structures from Morrison, Ross, and Kemp’s (2004) expanded performance-content matrix to identify the content structures. The following explains each of these structures.

Expanded Performance-Content Matrix. The matrix (Morrison et al., 2004) identifies six types of content. First are facts that are associations between two items. For example, C is the chemical symbol for carbon is a fact with an arbitrary association between C and carbon. Second are concepts that are categories of similar things. Gloves, doors, windows, and torts are examples of concepts that encompass similar things. The third category consists of rules and principles that express a relationship between concepts or direct behavior, such as prediction. Fourth are procedures that are a sequence of steps for accomplishing a goal, such as calculating the square footage of a room or how to apply varnish. The fifth category encompasses interpersonal skills, such as correct procedures for answering the telephone and communication with one or more other individuals. Sixth are attitudes which are a predisposition to respond to in a consistent manner (Fishbein & Ajzen, 1975), such as one’s attitude towards taking company office supplies.

Identifying Text Structures. The next step is to identify these six content structures in the content disassembly outline. This process requires the instructional designer to chunk the content and identify examples of these structures. When a content structure is found, it is labeled on the outline. After the disassembly of the content and identification of the content structures, an analysis of the instructional adequacy of the content is conducted.

The next step is to determine the instructional adequacy of the content for the distance education environment. To accomplish this task, we will use the design strategies from Morrison et al. (2004). Designers are not limited to this model for determining the instructional adequacy of the content. They can select any instructional design model or set of heuristics that provide detailed instructional strategies.

Before we can begin to determine the instructional adequacy of the content, we must define the objectives for the materials. These objectives can be stated as either behavioral or cognitive objectives.

In this section, we will describe the minimum content required to teach each of the structures (e.g., facts, concepts, principles, etc.). The requirements are divided into two categories (Morrison et al., 2004). The first structure is the presentation or initial presentation in which the content is presented to the learner. Second is a generative strategy or practice that helps the learners integrate the new information into their existing schema. The following paragraphs describe the minimum content requirement for each of the content types.

Facts. Facts are associations that are taught at the recall level. There are two minimum requirements for teaching a fact. First is a concrete representation of the fact through either direct experience, such as visiting a fire station or by stating an abstract fact such as “Nashville is the capital of Tennessee” by creating a map with a star indicating Nashville. Second is rehearsal and practice or a generative strategy such as a mnemonic device.

Concepts. The minimum requirements for teaching concepts are described by Tennyson and Cocchiarella (1986). First, the presentation of the concept must include the concept name, definition, and best example. Second, a gener ative strategy such as generating new examples and nonexamples is needed.

Principles and Rules. There are two approaches to presenting a principle or rule to a learner. First is the Eg-Rule approach that provides the learner with several examples and then asks the learner to state the rule. Second is Rule-Eg that presents the learner with the rule and then illustrates the rule with several examples (Markle, 1969). Examples of appropriate generative strategies needed to teach principles and rules include paraphrasing and elaboration strategies.

Procedures. Procedures are a series of ordered steps one must complete to do either a cognitive or psychomotor task. The presentation consists of a model of the performance, which could vary from a live or videotape demonstration to printed materials illustrating the steps with pictures and text. As an example, Sweller and Cooper’s (1985) worked examples are often used to model procedures. The generative strategy involves having the learner develop a mental model of the procedure and then practicing the procedure.

Interpersonal Skills. Interactions between two or more people are examples of interpersonal skills. Bandura’s (1977) social learning theory describes the four minimum requirements for instruction on interpersonal skills. First is providing the learner with a model of the interaction. This model might include a live or videotape example, a role-play, or case study. Second is encouraging the learners to develop a verbal and imaginal model of the interaction. Third is providing the students with a scenario for mental rehearsal. Fourth is overt practice through such activities as roleplaying or supervised interactions.

Attitudes. The minimum requirements for teaching attitudes are the same four criteria as for interpersonal skills: model, develop verbal and imaginal mode, mental rehearsal, and practice.

Determining the instructional adequacy of the content requires the evaluation of each content structure identified in the previous step. Using the minimum criteria stated in the previous paragraphs, the designer determines the adequacy of each occurrence of each content structure. Notes are made indicating any inadequacies in the presentation or generative strategy.

The Instructional Disassembler can also be used to evaluate existing instructional materials. For example, a designer who incorporates sharable content objects might use the Instructional Disassembler to analyze and evaluate an object. An instructional designer can use the same process to determine the adequacy of the content and the appropriateness of the instructional strategies. Evaluating instruction with the Instructional Disassembler is a three-step process. First, the instruction is disassembled into the smallest parts. Second, the content structures are identified in the disassembled content. Third, the instruction is evaluated for adequacy and appropriateness. The designer must use a design model as previously described to determine if minimum content to teach the content structures (e.g., fact, concept, principle, etc.) are present in the task analysis. Then, the instructional strategies for teaching the content are evaluated for accuracy against the prescriptions the designer has developed or that are provided as part of an instructional design model. Another application of this process is to construct metadata (“Scorm Best,” 2003) for sharable content objects. The analysis produced by the disassembler will identify the key points of the object that can then be use to create the metadata.

The results of the instructional disassembly form the basis for making instructional design decisions for the new delivery method. There are two approaches to improving the adequacy of the content and instruction.

First, if the designer has control of the original documents, that is, they were produced by an instructor, subject-matter expert (SME), or design team, then the documents can be modified. Additional content can be added to improve the content quality and appropriate instructional strategies can be integrated to improve the instructional adequacy of the material.

Second, if the content is part of a book or other copyrighted material that cannot be modified, then supplementary materials similar to a study guide are needed to improve the content and instructional adequacy of the materials. For example, a study guide approach might direct the students to read a few pages in the book. Or, an additional online book could be produced that summarizes the text information and includes discussion questions. Then, the learner’s attention is redirected to the study guide where additional content and/or instructional strategies are presented to supplement the book.

In this article we have presented a tool for instructional designers to use when determining if existing e-learning “courses” and materials or traditional courses that are to be ported to the Web are sound from an instructional design perspective. The tool allows the instructional designer to disassemble the content, identify the content structure, and analyze the adequacy of the content and instructional strategies. While this approach is detailed and time consuming, it provides a more thorough instructional design evaluation than a simple of review of the materials by subject-matter experts.

The authors would like to thank Katherine M. Kuhn and Howard Kalman for their comments on the manuscript.