IN SEARCH OF QUALITY: An Analysis of e-Learning Guidelines and Specifications Free

So, what is quality and how can you assure the quality of your instruction? There are two contrasting views: conventional and industrial. Most educators in K-12 and higher education take a conventional perspective, discussing quality in terms of effectiveness, efficiency, and appeal. Tactics focus on enhancing learner achievement, retaining a greater number of learners, and promoting learner satisfaction. In distinct contrast, industry increases quality by reducing variance around set standards. Figure 1 begins to compare the two views, illustrating how increases in quality differ over time.

Two distinct movements characterize the quest for quality in distance education that parallel the conventional and industrial view; the adoption of education guidelines and development of industry specifications. Both movements include input from, and serve the interests of, education, government, and industry. The goals of both movements are to assure that e-learning achieves its objectives dependably (for students and users) and efficiently (with minimal cost and maximum effectiveness). Again, the primary difference is that education guidelines focus on the quality of elearning courses and programs, whereas industry standards concentrate on the technical quality, reusability, and interoperability of learning objects. The difference in focus, in turn, has significant ramifications for those considering the adoption of standards and the development of a quality assurance system.

Traditional indicators, such as teacher credentials and time spent in class with a teacher, may be inappropriate for assuring the quality of distance learning programs. “New delivery systems test conventional assumptions, raising fresh question as to the essential nature and content of an educational experience and the resources required to support it” (Council of Regional Accrediting Commissions, C-RAC, 2000, p. iii). To address the proliferation of distance learning programs, professional organizations are publishing new guidelines that are being adopted by states, regional accrediting associations, K-12 schools, and institutions of higher education. This study analyzes six sets of guidelines, including:

1. The Council of Regional Accrediting Commissions (2000). Statement of the regional accrediting commissions on the evaluation of electronically offered degree and certificate programs (http://www.wiche.edu/telecom/Guidelines.htm)

2. The Institute for Higher Education Policy (2000). Quality on the line: Benchmarks for success in Internet-based distance education (www.ihep.com/quality.pdf)

3. The American Council on Education (ACE, 1997). Guiding Principles for Distance Learning in Learning Society (www. acenet.edu/calec/dist_learning/ dl_principlesIntro.cfm)

4. The American Distance Education Consortium (n.d.a., n.d.b). Guiding Principles for Distance Learning and Guiding Principles for Distance Teaching and Learning (www.adec.edu/admin/papers/ distance-teaching_principles.html).

5. The American Federation of Teachers (2000). Distance Education: Guidelines for Good Practice (www.aft.org/ higher_ed/downloadable/distance.pdf)

6. Open and Distance Learning Quality Council (ODLQC, 2001). Standards in open and distance education (http://www.odlqc.org.uk/st-int.htm).

The Best Practices generated by C-RAC (2000) seek to address concern that regional accreditation standards are not relevant to new distributed learning environments. Based on the Principles of Good Practice, initially drafted by the Western Cooperative for Educational Telecommunications (WCET, 1997), the Best Practices are meant to assist institutions in planning electronic distance education activities and provide a framework for selfassessment; they are not new evaluative criteria.

The National Education Association (NEA) and Blackboard Inc. jointly commissioned The Institute for Higher Education Policy (IHEP) to examine existing guidelines for distributed learning. An initial list of 45 benchmarks was then analyzed by faculty, administrators, and students from six colleges and universities. The final outcome of 24 benchmarks for success in Internet-based distance education was published by IHEP in 2000. Subsequently, 16 higher education leaders reviewed the IHEP benchmarks for a symposia sponsored by The Pew Learning and Technology Program, providing further insights into distance learning and quality assurance from a provider and consumer perspective (Twigg, 2001).

A national task force created by the American Council on Education and The Alliance: An Association for Alternative Programs for Adults generated The Guiding Principles for Distance Learning in a Learning Society (ACE, 1997) that focused on the changing nature of education and training, not on specific delivery systems or methods. These guidelines are neither a treatise nor a “how to” for distance learning; rather, they address the qualities of tomorrow’s future learning society. The purpose of the guidelines is to “help learners, educators, trainers, technologists, and accreditors/state regulators to develop, deliver, and assess formal learning opportunities” (Sullivan & Rocco, 1997).

The American Distance Education Consortium (ADEC) published the Principles for Distance Teaching and Learning (n.d.a) and the Principles for Distance Learning (n.d.b) to evaluate Web-based learning environments and nonformal educational programs. Like ACE, ADEC recognizes that Web-based instruction may be designed for distance and face-to-face learners. As such, the principles are foundational to high-quality learning environments in general, no matter where the learner resides. The Principles for Distance Teaching and Learning concentrates on course design and delivery. The Principles for Distance Learning addresses service and administrative policies.

The Guidelines for Good Practice, published by the American Federation of Teachers (2000), is based on a 1999 survey of 200 members. Although the focus is on 2-year, 4-year, and graduate credit-bearing degree programs, the guidelines are said to be applicable to all types of distance education, including job and skill training “because they are simply about good teaching” (AFT, 2000, p. 6). The guidelines are designed to help faculty negotiate distance education issues with management, as well as to help administrators and public officers who want to put quality at the center of their initiatives.

Recognizing that technology has changed the face of distance and open education, European accrediting agencies are also reviewing and adopted revised quality standards. Set up by the British government in 1968, the Open and Distance Learning Quality Council (ODL QC) operates as a voluntary distance learning registration system. Course providers must meet the Standards in Open and Distance Education published by ODL QC (2001) to register courses.

This study analyzes the six guidelines and extends findings from the Pew symposia (Twigg, 2001) by elaborating on instructional design issues and illustrating how industry specifications for the development, reusability, and interoperability of learning objects may be applied to complement education guidelines.

Analysis of the education guidelines reveals a number of important issues to consider if you are establishing a system to assure the quality of e-learning programs or courses.

In a review of standard-setting initiatives, members of the Pew Learning and Technology Program (Twigg, 2001) noted a high level of agreement among published guidelines. Table 1 illustrates that education guidelines posited by professional organizations center around five basic structures (i.e., institution, program, course, student support, and faculty support).

The convergence across guidelines suggests that those adopting quality standards should consider specifying guidelines for their institution, programs, courses, students, and faculty. However, there are a few points of divergence worth noting. Specifically, the guidelines proposed by C-RAC focus on program outcomes listed under Curriculum and Instruction and Evaluation and Assessment. They do not specify process-oriented standards associated with pedagogy and course design. In contrast, ODL QC specifies standards for course objectives, outcomes, and content, but do not offer guidelines for the design of programs or curriculum. Three sets of standards (i.e., ACE, 1997; ADEC, n.d.b; ODL QC, 2001) do not address faculty support, but all six sets recognize the importance of setting institutional and student support guidelines.

A closer look reveals greater congruence between guidelines than Table 1 suggests. Differences in guidelines may be more semantic than actual variations in implementation. For instance, ODL QC does consider program and curriculum issues, but from an institutional and a course perspective, addressing staffing and resource availability under guidelines for The Provider and degree and certification requirements under guidelines for Course Objectives and Outcomes. Similarly, C-RAC addresses course design issues, such as learning outcomes, instructor-learner interactions, and learner assessment, but at a programmatic level under the headings Curriculum and Instruction and Evaluation and Assessment, rather than specifying guidelines under Course Design or Pedagogical Guidelines.

The congruence across published guidelines suggests that there are several basic categories related to organizational structure. However, Table 1 demonstrates that comparable guidelines may be categorized in different ways. Current guidelines tend to mix the primary target of analysis. Some statements address the institution, while others focus on programs, and still others address courses and learning experiences. If you are adopting quality guidelines, consider establishing guidelines for your institution, programs, courses, students, and faculty, noting that it is not necessary to specify five categories, rather to address key issues related to each category. Also, keep in mind that “consistency and clarity concerning the organizational level being addressed would improve any statement about quality indicators in distance learning” (Twigg, 2001, p. 7).

Traditional criteria used to accredit degree and certificate programs have assumed that learning would take place if institutions provided certain inputs or resources (e.g., limited class size, full-time tenure-track faculty, student seat time, documented policies, equipped classrooms and libraries). Accrediting bodies, such as NCATE and SACS, are now placing emphasis on learning outcomes, giving institutions flexibility over how they achieve the outcomes. Advances in technology also challenge traditional views on what constitutes quality. Distance learning guidelines no longer focus on inputs. But, rather than concentrating on results, the guidelines center on process-oriented variables. For instance, the guidelines posited by IHEP (2000), AFT (2000), ACE (1997) and ADEC (n.d.a.) all specify requirements for learner engagement, media use, interactions, assessment, and feedback. Outcomes are specified by ACE, ADEC, and ODL QC (2001), but there is a definite attempt to guide the teaching and learning process.

For now, process guidelines may be warranted. Educators may not have the knowledge necessary to define their own methods for creating e-learning materials or facilitating the elearning process. A balance between input-, process-, and outcome-oriented guidelines may be useful for assuring the quality of elearning experience, at least for the immediate and near future.

To set quality guidelines, it is important to consider the level of organizational effort you want to measure. The Organizational Elements Model (OEM) (Kaufman & Hirumi, 1992; Kaufman & Watkins, 2000; Kaufman, Watkins, & Leigh, 2001) provides a useful framework for distinguishing different levels of organizational effort (Table 2).

Application of the OEM to e-learning illustrates how standards can be written for varying levels of organizational effort. Table 2 notes examples of inputs, processes, and three levels of results (i.e., products, outputs, and outcomes) relative to e-learning. If you are adopting standards, it is important to decide if you are going to specify guidelines for inputs, processes, products, outputs, and/or outcomes. Again, the key is the consistency and clarity with which guidelines are communicated and addressed.

Higher education leaders have expressed concern that published guidelines appear more like statements of adequate rather than best practice (Twigg, 2001). “Statements [of quality] do not say, ‘you should have these outcomes.’ They say only, ‘you should have outcomes’” (Twigg, 2001, p. 17). In other words, published guidelines do not always specify the level at which distance educators, learners, or organizations are expected to perform. For example, IHEP (2000) benchmarks state that students should (a) receive information about programs (e.g., admissions, tuition, fees, books, supplies, technical and proctoring, services), (b) receive hands-on training and information for securing instructional materials, and (c) have access to technical assistance throughout course/program. They do not specify the level at which educators should perform each of these functions. Compare such statements to a sample of student support standards published by Athabasca University (Abrioux, 2002) (Table 3). The differences between minimum requirements and high-quality standards are apparent. Athabasca not only specifies the services to be offered, but also defines performance standards and lists contact information for each service.

World-class benchmarks not only define what should be done but also delineate how well it should be done. When specifying quality standards, you should consider whether you want to define minimum requirements or identify world-class benchmarks that people within your organization should strive to achieve.

The published guidelines do address important instructional variables, such as objectives, content, assessment, feedback, and media use. However, they ignore a number of evolving pedagogical and instructional design principles. For example, members of the IHEP (2000) study excluded 19 of the original 45 benchmarks because they felt they were not necessary for assuring quality. Pew symposium members acknowledged that the excluded benchmarks may not be essential, but also noted that several were “ones that lead to higher quality practices because they were more learner-centered and incorporated pedagogical approaches of proven effectiveness” (Twigg, 2001, p. 8). The following design principles are not addressed by published guidelines. To establish world-class benchmarks, consider specifying guidelines for:

1. The alignment of objectives and assessments. Have you ever taken a test and wondered, where the #&%@ did that question come from? Alignment between explicit objectives and criteria is fundamental to high-quality instruction (Berge, 2002; Dick, Carey, & Carey, 2005). If an objective states that students will be able to list key concepts, assessments should ask students to list key concepts. If an objective states that learners will be able to analyze a case, the assessment should ask learners to analyze a case. High-quality learning environments present learners with explicit and congruent learning objectives and assessment criteria. To establish world-class guidelines, consider specifying the alignment of learner assessments with objectives.

2. The alignment of objectives and instructional events. Research suggests that how we teach should be based on what we teach. The methods used to teach verbal information should differ from the methods used to teach a procedure that, in turn, should differ from the methods to teach complex problem solving, and so forth. Smith and Ragan (1999) classify alternative instructional events that have been found to facilitate achievement of various learning outcomes (Table 4).

   TABLE 4

   Grounded Events Related to Learning Outcomes

   Learning Outcome	Grounded Event
   Verbal information	Mneumonics and Metaphoric Devices Instructor or learner generated images Rehearsal Clustering and chunking into categories Expository and narrative structures Graphic and advanced organizers Write meaningful sentences Devise rule
   Concepts	Inquiry and Expository Approaches Attribute Isolation Concept Trees Analogies, Mnemonics and Imagery
   Rules	Determine if the procedure is required. List the steps in a procedure. Complete the steps in a procedure. Elaborate sequence Check appropriateness of completed procedure.
   Problem solving	Presentation of the Problem Analyze Problem Space Apply Appropriate Principles Practice
   Cognitive strategies	Discovery and Guided Discovery Observation Guided Participation Direct Instruction
   Attitudes	Demonstrate desired behaviors Practice desired behaviors Provide reinforcement for the desired behavior Communicate persuasive messages from highly credible sources Create dissonance
   Psychomotor skills	Massed versus Spaced Practice Whole versus Parts Practice Progressive parts practice Backwards chaining

   Learning Outcome	Grounded Event
   Verbal information	Mneumonics and Metaphoric Devices Instructor or learner generated images Rehearsal Clustering and chunking into categories Expository and narrative structures Graphic and advanced organizers Write meaningful sentences Devise rule
   Concepts	Inquiry and Expository Approaches Attribute Isolation Concept Trees Analogies, Mnemonics and Imagery
   Rules	Determine if the procedure is required. List the steps in a procedure. Complete the steps in a procedure. Elaborate sequence Check appropriateness of completed procedure.
   Problem solving	Presentation of the Problem Analyze Problem Space Apply Appropriate Principles Practice
   Cognitive strategies	Discovery and Guided Discovery Observation Guided Participation Direct Instruction
   Attitudes	Demonstrate desired behaviors Practice desired behaviors Provide reinforcement for the desired behavior Communicate persuasive messages from highly credible sources Create dissonance
   Psychomotor skills	Massed versus Spaced Practice Whole versus Parts Practice Progressive parts practice Backwards chaining

   View large

   High-quality learning environments present learners with instructional events based on targeted learning outcomes. To establish world-class guidelines, consider specifying the need to incorporate instructional events that are designed to facilitate the achievement of learning outcomes based on research and theory.

3. The nature of feedback. Feedback is vital to e-learning. At minimum, feedback is essential for closing message loops (Northrup & Rasmussen, 2000; Yacci, 2000), informing learners that communications are complete (Berge, 1999; Liaw & Huang, 2000; and Weller, 1988, as cited by Northrup, 2001). Feedback may also (a) increase response rates or accuracy, (b) reinforce correct responses to prior stimuli, and (c) change erroneous responses (Kulhavy & Wager, 1993). Feedback comes in two basic forms; confirmatory and corrective. Confirmatory feedback lets students know what they did correctly. Corrective feedback identifies areas and provides recommendations for improvement. Current guidelines recognize the importance of providing timely and appropriate feedback, but they do not detail the nature of the feedback. To develop world-class guidelines, consider delineating what is meant by timely and appropriate feedback based on research, theory, and documented best practices.

4. The design and sequencing of e-learning interactions. In traditional classroom settings, key interactions that affect learner attitudes and performance often occur spontaneously, in real-time. Good instructors interpret verbal and nonverbal cues, clarify expectations, facilitate activities, promote discussions, elaborate concepts, render guidance, and provide timely and appropriate feedback as they present content in a clear and engaging manner. Good instructors can also make up for flaws in design by utilizing their charisma to gain and sustain learners’ attention and their experience to shed light on complex or confusing content matter. During e-learning, opportunities to interact in “realtime” are relatively confined. Key interactions that occur spontaneously in traditional classroom environments must be planned and managed as an integral part of e-learning. Hirumi (2002a, 2002b) posits several grounded instructional strategies to guide the design and sequencing of e-learning (Table 5).

   TABLE 5

   Sample of Grounded Instructional Strategies

   Nine Events of Instruction	Student-Center Learning	Jurisprudential Inquiry
   Gain Attention Inform Learner of Objective(s) Stimulate Recall of Prior Knowledge Present Stimulus Materials Provide Learning Guidance Elicit Performance Provide Feedback Assess Performance Enhance Retention and Transfer	Set Learning Challenge Negotiate Learning Goals and Objectives Negotiate Learning Strategy Construct Knowledge Negotiate Performance Criteria Assess Learning Provide Feedback (Steps 1-6) Communicate Results	Orientation to the Case Identifying the Issues Taking Positions Exploring the Stance(s), Patters of Argumentation Refining and Qualifying the Positions Testing Factual Assumptions Behind Qualified Positions
   Simulation Model	Direct Instruction	Experiential Learning
   Orientation Present topic of simulation Explain simulation Give overview Participant Training Set-up scenario Assign roles Hold abbreviated practice Simulation Operations Conduct activity Feedback and evaluation Clarify misconceptions Continue simulation Participant Debriefing Summarize events Summarize difficulties Analyze process Compare to the real world Appraise and redesign the simulation	Orientation Establish lesson content Review previous learning Establish lesson objectives Establish lesson procedures Presentation Explain new concept or skill Provide visual representation Check for understanding Structured Practice Lead group through practice Students respond Provide corrective feedback Guided Practice Practice semi-independently Circulate, monitor practice Provide feedback Independent Practice Practice independently Provide delayed feedback	Experience—bImmerse learner in “authentic” experience. Publish—Talking or writing about experience. Sharing thoughts and feelings. Process—Debrief: Interpret published information, defining patterns, discrepancies and overall dynamics. Internalize—Private process, learner reflects on lessons learned and requirements for future learning. Generalize—Develop hypotheses, form generalizations and reach conclusions. Apply—Use information and knowledge gained from lesson to make decisions and solve problems.
   Inquiry Learning	Inductive Thinking	Problem-Based Learning
   Confrontation with the Problem Explain inquiry procedures Present discrepant event Data Gathering—Verification Verify nature of objects and conditions Verify the occurrence of the problem situation Data Gathering—Experimentation Isolate relevant variables Hypothesize and test casual relationships Organizing, Formulating and Explanation—Formulate rules or explanations Analysis of inquiry process—Analyze inquiry strategy and develop more effective ones.	Concept Formation Enumeration and listing Grouping Labeling, Categorizing Interpretation of Data Identify critical relationships Explore relationships Make inferences Application of Principles Predicting consequences Explaining predictions Verifying predictions	Starting a New Problem Set problem Describe requirements Assign tasks Reason through the problem Commitment to outcome Shape issues and assignment Identify resource Schedule follow-up Problem Follow-Up Resources used Reassess the problem Performance Presentation(s) After Conclusion of Problem Knowledge abstraction and summary Self-evaluation

   Nine Events of Instruction	Student-Center Learning	Jurisprudential Inquiry
   Gain Attention Inform Learner of Objective(s) Stimulate Recall of Prior Knowledge Present Stimulus Materials Provide Learning Guidance Elicit Performance Provide Feedback Assess Performance Enhance Retention and Transfer	Set Learning Challenge Negotiate Learning Goals and Objectives Negotiate Learning Strategy Construct Knowledge Negotiate Performance Criteria Assess Learning Provide Feedback (Steps 1-6) Communicate Results	Orientation to the Case Identifying the Issues Taking Positions Exploring the Stance(s), Patters of Argumentation Refining and Qualifying the Positions Testing Factual Assumptions Behind Qualified Positions
   Simulation Model	Direct Instruction	Experiential Learning
   Orientation Present topic of simulation Explain simulation Give overview Participant Training Set-up scenario Assign roles Hold abbreviated practice Simulation Operations Conduct activity Feedback and evaluation Clarify misconceptions Continue simulation Participant Debriefing Summarize events Summarize difficulties Analyze process Compare to the real world Appraise and redesign the simulation	Orientation Establish lesson content Review previous learning Establish lesson objectives Establish lesson procedures Presentation Explain new concept or skill Provide visual representation Check for understanding Structured Practice Lead group through practice Students respond Provide corrective feedback Guided Practice Practice semi-independently Circulate, monitor practice Provide feedback Independent Practice Practice independently Provide delayed feedback	Experience—bImmerse learner in “authentic” experience. Publish—Talking or writing about experience. Sharing thoughts and feelings. Process—Debrief: Interpret published information, defining patterns, discrepancies and overall dynamics. Internalize—Private process, learner reflects on lessons learned and requirements for future learning. Generalize—Develop hypotheses, form generalizations and reach conclusions. Apply—Use information and knowledge gained from lesson to make decisions and solve problems.
   Inquiry Learning	Inductive Thinking	Problem-Based Learning
   Confrontation with the Problem Explain inquiry procedures Present discrepant event Data Gathering—Verification Verify nature of objects and conditions Verify the occurrence of the problem situation Data Gathering—Experimentation Isolate relevant variables Hypothesize and test casual relationships Organizing, Formulating and Explanation—Formulate rules or explanations Analysis of inquiry process—Analyze inquiry strategy and develop more effective ones.	Concept Formation Enumeration and listing Grouping Labeling, Categorizing Interpretation of Data Identify critical relationships Explore relationships Make inferences Application of Principles Predicting consequences Explaining predictions Verifying predictions	Starting a New Problem Set problem Describe requirements Assign tasks Reason through the problem Commitment to outcome Shape issues and assignment Identify resource Schedule follow-up Problem Follow-Up Resources used Reassess the problem Performance Presentation(s) After Conclusion of Problem Knowledge abstraction and summary Self-evaluation

   View large

   To establish world-class standards, consider specifying the need to design and sequence e-learning interactions based on grounded instructional strategies.

5. Motivational design. Educators recognize that motivation is essential to student learning. Students must be presented with the appropriate skills and knowledge and they must be motivated to learn and use them. Even though there are numerous theories of human motivation, published guidelines do not address motivational factors, at least not with the precision they attend to concept acquisition. Keller (1987a, 1987b) presents a systematic process for designing motivationally effective instruction that subsumes related theories. In short, Keller’s ARCS model suggests that, to motivate students to learn, instruction must (a) gain and sustain learners’ Attention, (b) be Relevant to learners’ needs and interest, (c) promote learners’ Confidence in their ability to succeed, and (d) ensure that learners are Satisfied that their efforts were worthwhile. Table 6 depicts some of the tactics educators can apply to meet motivational quality guidelines.

   TABLE 6

   Tactics for Motivating Students to Learn

   Motivational Constructs	Motivational Design Tactics
   Attention	Perceptual Arousal—Stimulate senses Inquiry Arousal—Stimulate curiosity Variability—Vary stimulus
   Relevance	Goal Orientation—Help students create and achieve goals Motive Matching—Address specific needs Familiarity—Relate to learners’ past experiences
   Confidence	Requirements—Awareness of expectations and evaluation criteria Success Opportunities—Opportunities to experience success Personal Control—Link success or failure to student effort and abilities
   Satisfaction	Natural Consequences—Meaningful opportunities to apply skills Positive Consequences—Positive reinforcement Equity—Consequences perceived to be fair by all students

   Motivational Constructs	Motivational Design Tactics
   Attention	Perceptual Arousal—Stimulate senses Inquiry Arousal—Stimulate curiosity Variability—Vary stimulus
   Relevance	Goal Orientation—Help students create and achieve goals Motive Matching—Address specific needs Familiarity—Relate to learners’ past experiences
   Confidence	Requirements—Awareness of expectations and evaluation criteria Success Opportunities—Opportunities to experience success Personal Control—Link success or failure to student effort and abilities
   Satisfaction	Natural Consequences—Meaningful opportunities to apply skills Positive Consequences—Positive reinforcement Equity—Consequences perceived to be fair by all students

   View large

   To define world-class guidelines, consider learner motivation, using insights derived from research on instructional design (such as the ARCS model) to guide the development of related standards.

The development of high-quality learning materials requires considerable resources. To optimize return on investment, the materials must be reusable. In other words, educators must be able to use the materials multiple times in alternative contexts. The concept of reusing educational resources is not new. There are extensive databases housing lesson plans and distance learning initiatives that share both human and material resources. To facilitate the reusability of e-learning materials, the documents or files must be interoperable. In other words, educators must be able to readily transfer e-learning materials (or objects) across settings.

Education guidelines focus on pedagogical aspects of e-learning. Guidelines are written primarily to ensure the quality of educational programs and courses by concentrating on the teaching and learning process and the provision of academic services. Few guidelines are written to facilitate the sharing and reuse of materials. To garner a significant return on the investment in the development of high quality e-learning materials, consider adopting industry specifications to promote technical quality. In other words, to ensure that learning objects meet technical specification to enable the interoperability and reusability of the objects).

When considering industry approaches to quality, keep in mind that, in the strictest sense, standards can only come from accredited bodies, such as the Institute of Electrical and Electronics Engineers (IEEE) and the International Standards Organization (ISO). Many so-called “standards” are actually guidelines, specifications, or statements of good practice. In short, a standard is a sanctioned specification. A standards body must determine that a given specification meets broad, industry-wide needs before it can become a standard. None of the criteria analyzed in this article are true standards, although the Sharable Content Object Reference Model (SCORM) is on a fast track toward becoming an industry standard.

Industry specifications for e-learning are related to, yet distinct from, education guidelines. In industry, specifications for learning objects are set to assure quality, but from a technical rather than a pedagogical perspective. This is not to say the industry is not concerned with pedagogy. In fact, there is a great deal of work going on throughout industry to establish the “next generation” of e-learning, in which the educational quality of content is being addressed along multiple fronts (e.g., individualized learning, problem-based learning, learning-by-doing, etc.). However, such efforts are seen as instructional design issues and are not treated by the formation of technical specifications for guiding the development of e-learning resources.

High-quality instruction achieves its objectives dependably (for all students and users) and efficiently (with minimal cost and maximum effectiveness) (Advanced Distributed Learning, 2002). Industry specifications focus on facilitating development (efficiency) and minimizing cost. To better understand industry specifications, it may be useful to first review several fundamental assumptions and the basic premise for the development of reusable learning objects.

It is assumed that there are thousands of colleges, universities, and public schools that teach the same subjects, such as introductory biology. It is also assumed that many of these introductory courses cover similar topics, such as cell division, and that the properties of cell division remain fairly constant across institutions. As a result, we have thousands of similar lessons about cell division. Now, assume that each institution puts its introductory course online. Do we need thousands of similar online lessons on cell division? The basic premise behind the development of learning objects is that the world needs a few representations of basic instructional units.

Let’s assume that it takes $1,000 to produce an interactive, multimedia instructional unit on cell division. If 1,000 institutions pay $1,000 to generate their own unit, it would result in a total expenditure of $1,000,000. In comparison, if the institutions shared development costs, the cell division unit would cost $1 per institution. The economic reasons for developing learning objects and establishing standards to ensure technical quality and interoperability are convincing. History also suggests that significant benefits are not realized without the widespread adoption of common standards (e.g., standard voltage and plugs for electricity; standard gauges for railroad track; common TCP/IP, http, and HTML standards for the Internet). The challenge lies in establishing useful standards and relatively seamless processes that can be readily adopted and maintained by a critical mass of people.

Industry specifications published by professional organization and analyzed in this study include:

1. The Instructional Management Systems Project (IMS, 2000). Open specifications for facilitating online distributed learning activities (www.imsproject.org/).

2. The Advanced Distributed Learning Initiative (ADL, 2002). Sharable Content Object Reference Model (SCORM). (www.adlnet.org)

3. The Institute of Electrical and Electronics Engineers (IEEE, 2002). Learning Technology Standards Committee (LTSC) P1484 (itsc.ieee.org/).

4. The Aviation Industry Computer-Based Training Committee (AICC, 1999). AICC Guidelines and Recommendation for Web-based Computer Managed Instruction (AGR-010 (www.aicc.org/)

5. ARIADNE (2002). ARIADNE Educational Metadata Recommendation—V3.2. (www.ariadne-eu.org).

6. Dublin Core Metadata Initiative (DCMI, 2002). Dublin Core Metadata Element Set Version 1.1. (www.dublincore.org).

Like education guidelines, industry specifications are not born at once; they emerge through consensus from related initiatives. For instance, the Instructional Management Systems (IMS) Global Learning Consortium is a coalition of academic, commercial, and government organizations promoting the development of open specifications for online learning activities. The goals of the IMS are similar to other industry initiatives: to define technical specifications for the interoperability of applications and services in distributed learning and to support the incorporation of specifications into products and services worldwide.

The Advanced Distributed Learning (ADL) Initiative, supported by the U.S. Government, released the Sharable Content Object Reference Model (SCORM) that is viewed as one of the best and most recent applications of elearning specifications (Hodgins & Conner, 2000). According to Walker (president of the IMS consortium), ADL is not trying to dictate the definition of specification. Rather, it takes the specifications identified by other standards bodies into a test-bed and evaluates whether they have the desired effects (Barron, 2000).

According to Hodgins and Conner (2000), most groups creating learning specifications use the Institute of Electrical and Electronics Engineers (IEEE) Learning Technology Standards Committee (LTSC) P1484 to cover topics including object metadata, student profiles, course sequencing, computer managed instruction, competency definitions, localization, and content packaging. As one of the most recognized standards bodies, IEEE LTSC has initiated work toward establishing full International Standards Organization (ISO) standards for learning technology.

The Aviation Industry Computer-Based Training Committee (AICC) is an international association of technology-based training professionals. The AICC develops guidelines to (a) promote the implementation of CBT among airplane operators, (b) develop guidelines for interoperability, and (c) provide an open forum for discussing CBT and other training technologies. Although AICC focuses on the aviation industry, technology-based training and computer software and hardware vendors are adapting AICC guidelines for their own industries.

Comprised of partners from Belgium, Finland, France, Italy, Spain, Switzerland, the Netherlands, and the United Kingdom, and supported by the European Union Commission and the Swiss Federal Office for Education and Science, ARIADNE seeks to apply IEEE LTSC specifications for metadata in a European context. ARIADNE’s goal is to develop an international system of interconnected knowledge pools (KPS) and to develop tools and basic methods for maintaining and exploiting the KPS.

The Dublin Core Metadata Initiative (DCMI) is also an open forum engaged in the development of interoperable metadata standards that have been adopted in Australia, Canada, Denmark, Finland, Ireland, and the United Kingdom. Originally conceived to describe Web resources, the DCMI has attracted the attention of museums, libraries, government agencies, and commercial organizations that often search for and access electronic resources. In December 2000, the IEEE LTSC on Learning Objects Metadata and the DCMI signed a Memorandum of Understanding announcing a joint commitment to the development of interoperable metadata for learning, education, and training.

Analysis of the industry specifications reveals important variables to consider if you are establishing a system to assure the quality of elearning programs, courses, or objects.

Advances in technology and increased interest in just-in-time training has resulted in the disaggregation of content into smaller instructional units. Working professionals are now less likely to go to universities to complete courses and degree programs. The emphasis is on providing smaller chunks of instruction (referred to as learning objects) at the moment and location of need through the use of modern telecommunication technologies.

Brennan, Funke, and Andersen (2001) define a learning object as the smallest standalone piece of instruction that contains an objective, an activity, and an assessment, wrapped by descriptive metadata. Metadata describes the nature and purpose of each object and is used to index the objects so that others can search for, retrieve, and reuse the object. The nature of the objectives, activities, and assessments that comprise a learning object is still under debate. Adherence to specifications, however, should allow objects to be shared across platforms, increasing the efficiency of the production process.

Table 7 lists six sets of specifications published by professional organizations for assuring the technical quality and interoperability of learning objects. Clearly, industry specifications differ from education guidelines. The specifications are technical in nature, addressing variables such as the XML binding, metadata coding, and the packaging of learning objects, rather than the design and delivery of courses and programs.

A fundamental question must be answered: “Are you going to set quality guidelines for programs, courses, faculty, and students, and/ or meet specifications for the storage and interoperability of learning objects?” The ramifications are considerable. Not only does the approach used to create and store courses differ significantly from those used to generate and manage objects, the purpose and nature of the quality assurance system also differ, as illustrated by a comparison of Tables 1 and 7.

Similar to education guidelines that deal with different organizational structures, industry specifications address different aspects of an e-learning system. Figure 2 illustrates how SCORM v1.2 incorporates existing IMS, AICC, and IEEE specifications. The interconnectedness of the published specifications is further demonstrated by the SCORM metadata specifications that are based on IEEE Learning Object Metadata specifications that were developed as a joint effort between the IMS and ARIADNE.

SCORM differs from other published specifications in that SCORM addresses a complete learning system. IMS and ADL share the architecture embodied in SCORM, but focus only on components of an entire system. According to Walker (Barron, 2000), SCORM is applicable to large corporations, government agencies, or universities striving to establish a comprehensive approach to training and education. The focus of IMS and other standards organizations, such as the IEEE, is on specifications that may be used by everyone.

If you decide to adopt specifications for elearning objects, you must decide if you want to define specifications for the entire e-learning system (like those specified by SCORM) or for components of your e-learning system. You can concentrate on storing and accessing objects, meeting specifications for metadata (like ARIADNE and the DCMI), or you can adopt specifications for various aspects of an e-learning system (like IEEE and IMS), or you can meet specifications for aggregating content and creating a run-time environment (like the ADL initiative). If you go beyond the specification of metadata, additional human resources (or at least a redefinition or reclassification of job descriptions) may be required. Someone well versed in programming is nec essary to apply and stay abreast of industry specifications.

Metadata recognizes that learning objects consist of pedagogical attributes. However, there is considerable debate as to what constitutes an object. Popular definitions posit that learning objects contain a measurable objective, an activity, and an assessment that are classified by metadata (e.g., Brennan et al., 2001). Other definitions quantify the size of the object using metrics such as duration (e.g., no more than 30 seconds to review) or the amount of information (e.g., no more than three individual screens of information). Recognizing that optimal object size depends on many factors, standards organizations neither specify the size nor composition of learning objects. Rather, they opt for broad definitions that approach size and composition based on need. For example, the IEEE (2002) defines a learning object as “any entity, digital or nondigital, that may be used for learning, education or training” (p. 5).

Such definitions are problematic. As Merrill (cited in Welsch, 2002) has noted, “If everything is an object, then nothing is a learning object” (p. 17). On an interpersonal level, the lack of a commonly-accepted definition can lead to confusion and miscommunications among experts and development team members. From an operational perspective, the lack of consensus makes it difficult to classify and assemble objects in a pedagogically sound fashion.

From a designer’s perspective, failure to define the specific elements of learning objects makes it difficult to code objects. When a designer classifies an object, does he or she consider an entire instructional unit as an object or specific elements of a unit as objects? Is a graphic an object? How about an activity? Is a set of objectives an object or must an object contain an objective, an activity, and an assessment? Some specifications even refer to learning assets (e.g., ADL, 2002) that may or may not be considered elements of a learning object.

Educators who seek to combine objects to create instructional modules or courses may neither have the time nor the expertise necessary to combine the objects in a cohesive manner. Let’s say a biology teacher finds an excellent object that depicts cell division, but finds the object does not contain appropriate objectives or assessments. Will he or she have the time and knowledge necessary to add the key instructional elements? If not, learners who access objects may find them lacking and fail to achieve specified outcomes. Unless the size—or, more specifically, the composition of a learning object—is further delineated, it may be difficult, if not impossible, to realize the long-term vision of the ADL initiative: to create a system that will store sharable content objects from across the Web and assemble them in real-time to provide learning and assistance anywhere, at anytime (Figure 3).

If you consider adopting industry specifications for learning objects, you must keep in mind objects may vary in size and composition. Additional guidelines and/or specifications may be necessary to assure the pedagogical effectiveness of e-learning courses and programs that are based on learning objects.

Like education guidelines, industry specifications do not to address key principles derived from research and theory. Although considerable progress has been made to formalize the technical aspects of industry specifications (e.g., to assure interoperability), that does not mean that the objects are pedagogically sound. “The interoperability that they (ADL) are promoting is just plumbing, and they don’t care what goes into the pipes. It could be spring water, or it could be sewage” (Anderson, cited in Welsch, 2002, p. 17).

Recognizing that existing specifications do not necessarily assure the pedagogical quality of instruction, the IMS (2000) is in the process of defining standards for Learning Design and Simple Sequencing. The ADL (2002) is also developing specifications for Learning Information Profiles to account for differences in learner characteristics, and generating additional specifications for delineating learner assessments. It will be interesting to see if the new specifications address the pedagogical concerns expressed in this article.

Earlier, it was noted that education guidelines do not address important pedagogical principles related to alignment, feedback, sequencing, and motivation. Analysis of industry specifications yield similar concerns, but applied to learning objects and assets rather than academic courses and programs. Research and theory provide an empirical foundation for defining such specifications. The challenge lies in distilling and applying the plethora of findings in a manner that is specific enough to assure the pedagogical effectiveness of the instruction, yet simple enough to follow and flexible enough to promote creativity while accounting for differences in learning outcomes, learner characteristics, and educational philosophies.

Renewed interest in distance education has given us an unprecedented opportunity to reflect on current and past practices and establish a dialog among practitioners and researchers to guide future activities. It has stimulated discussions on pedagogy, instructional design, and the use of technology across disciplines rarely seen in industry or academia. New initiatives have resulted in products that not only facilitate distance learning, but learning in traditional and hybrid settings. Return on investment is not limited to distance learners; significant benefits are also accrued by on-campus students as educators transfer skills and educational materials to enhance learning in alternative settings. However, benefits to both distant and on-campus students cannot be realized without the development and delivery of quality programs, courses, and/or objects.

To help ensure quality, one of the first issues that must be addressed is whether your organization wants to take an educational or industrial approach. In other words, will your organization define guidelines for programs and courses, or for learning objects and assets? Or, is a combination of approaches more appropriate? After establishing the basic approach, you must also determine which set of guidelines or specifications to adopt and/or adapt. This article identifies some of the key factors to consider as you answer these basic questions. However you proceed, there is still one fundamental question that must be answered that transcends both education guidelines and industry specifications; that is, will you define separate standards for distance learning and traditional classroom courses?

In the belief that new delivery systems require new standards, institutions and accrediting bodies have generated different requirements for assessing the quality of distance education and traditional classroom instruction. Certainly, from a logistical perspective, applying two sets of standards to assure the quality of instruction takes more time and money than one set and complicates the certification process. From an operational perspective, the increasing number of hybrid courses that combine distance and face-to-face components bring to question the validity of either set of requirements. Should a third set of standards be developed, is either set more appropriate or should we consider defining one set of standards for teaching and learning that transcends the method used to deliver the instruction?

The guiding principles posited by the American Council on Education and The Alliance (ACE, 1997) represent a movement toward outcomes or results-orientated guidelines that do not distinguish between delivery system. Rather than specifying how distance learning should take place, the guiding principles seek to address, “the qualities of tomorrow’s future learning society” (Sullivan & Rocco, 1997). In the immediate future, the complete dissolution of traditional standards and the adoption of new standards may be too overwhelming for many. Maybe it is more practical to define separate standards first, then look to synthesize them over time. What we do know is that advances in technology will continue to outpace research and challenge conventional views of teaching and learning. The publication of guidelines and specifications is seen as a solid step toward assuring the quality of e-learning, but as the analysis reveals, our work is far from finished. The search for quality continues, and those who seek it may find the conversations resulting from the quest as— or even more—valuable than the actual adoption of quality standards.