This article addresses how evaluation criteria improve educational Web-information system design, and the tangible and intangible benefits of using evaluation criteria, when implemented in an educational Web-information system design. The evaluation criteria were developed by the authors through a content validation study applicable to multidimensional scaling (MDS). The article outlines an optimal educational Web-information system using the underlying dimensions. Included is a definition and discussion of an educational Web-information system consisting of cognitive, affective, and social categories. The best educational Web-information systems maximize the accessibility of quality information through the adaptation of user needs.
Background and Introduction
The modern society chooses the digital network as the mode of storing, interchanging, interpreting, and representing philosophy, ideology, or information. The information system is a system functioning for the collection, processing, storage, transmission, display, dissemination, and disposition of information. Information and the individual are independent within a distance and the distance can be narrowed while an individual is interacting with the information through the medium of an application system—infrastructure, interface, and service. Thus, the components of the information system include the user of the information system, information, and application system—namely, technology including infrastructure—the computer-human interface (interaction), and the invisible services.
Canal (2004) argued that an information system was divided into two domains—structure and behavior. The domain structure related to computer technical methods has three elements: a repository that stores data; an interface where the user, information, and infrastructure interact; and a channel that physically links data. The other domain is behavior, which is the attitude of the stakeholder, designer, or owner of the information system. In the behavior domain, the values the information system pursues are represented through messages while the user interacts with the technology.
The Alliance for Telecommunications Industry Solutions (2006) argued that standards would contribute to enhancing reliability and validity, which are criteria of the quality of an information system. The contributions and requirements of standards are to:
Identify and define performance parameters and levels for the speed, accuracy, dependability, availability, and robustness of connection establishment, information transfer, and connection disengagement;
Define measurement techniques for these performance parameters;
Define methods for characterizing network and signal processing performance for customer applications;
Develop transmission planning guidance for the deployment of signal processing devices such as echo cancellers and VoIP elements; and
Consider the characteristics of signal processing and multimedia systems and the needed interworking among network technologies and services such as IP, Frame Relay, ATM, SONET, OTN, TDM, Wireless, etc (Alliance for Telecommunications Industry Solutions, 2006, ¶ 3)
The Information System
Regardless of the representation mode of information, electronic or physical, an information system is a systematized method to collect, archive, and serve data or resources. Taking this into consideration, society selects the best structure of information system fitting its technology and culture. For example, in Eco’s novel The Name of the Rose, an Italian monastery had the largest library, which was the best information system in the fourteenth century. Today, the information system is wherever the Internet is connected. Using today’s modern information system, the monks would not be limited to the confines of their monastery for their scholarly endeavors. One example of a modern information system is the digital library.
The Digital Library
The digital library is an electronic information space that includes all ideals of e-learning. The ideal digital library provides universal access to all forms of information service, including digital modes. It strives to facilitate collaboration among people from various fields, such as business, politics, education, and research, and contribute to learning experience continuing for a lifetime (Association of Research Libraries, 1995).
Traditional libraries focused on holdings and effective ways to deliver learning materials that were not provided. However, the Internet has changed the delivery modes of learning materials and the digital library has focused on “access” (Hughes, 2004, p. 376). Greenstein (2000) defined a digital library service as a “networked online information space in which users can discover, locate, acquire access to and, increasingly, use information” (p. 290). The digital library supports different perspectives from various cultures and strengthens its multiple functions including providing “lifelong innovative, scholastic research and lifelong learning. It is designed for the library’s patrons as well as for its professional staff and with an eye on the needs and capacities of those who supply it with information content” (Greenstein, 2000, p. 290). Matson and Bonski (1997) demonstrated some characteristics of the digital library. They included computer technology related data collection, a standardized part of “the emerging national information infrastructure,” “information products” from “online database,” and electronic “library systems” (¶ 5).
The rationale for digital library projects posited that these initiatives improved quality learning and assisted in obtaining the learning goals. For example, the University of Michigan Digital Library Project, formed in 1994, implemented the open and redistribution of information through the Web. The project also demonstrated that it increased the opportunities for inquiry learning (The Digital Library of the University of Michigan, 2005). A case study of the Alexandria Digital Earth Prototype at the University of California, Los Angeles, and the University of California, Santa Barbara posited that the digital library lead to better gains in students’ learning and contributed to the development of learners’ scientific thinking (Borgman et al., 2000).
The digital library is the transformation of the traditional library, which includes broader and more integrated information, perspectives and benefits to more diverse users (Hughes, 2004). Borgman et al. (2000) asserted that the digital library should meet the users’ needs in terms of information process including collecting, systematizing, and examining knowledge and information. It also should be able to deal with questions from the users. Hughes (2004) also posited that the digital library should:
be easily found among other institutional Web pages;
provide an up-front tutorial for the new leaner;
be integrated with the institution’s online courses;
provide tools to assist with online searches; and
provide access to personal assistance, if needed (p. 376).
Schooling and the Digital World
Another example of changing culture into digital world is schooling. In the movie Dead Poets Society (Haft, Witt, Thomas, & Weir, 1989), a gifted teacher was assigned about 20 intelligent students from wealthy families in a classic traditional English high school. The gifted teacher inspired the students by interacting with them, face to face, in an attractive 1959 prep school. However, in today’s digital society 50 years later, the same prep school might choose virtual schooling, or e-learning, instead of instruction in a traditional brick and mortar structure. Similar to the digital library, e-learning is one of the latest emerging types of education supported by technology (Lezberg, 1998) in order to accommodate to social needs (Gilbert & Driscoll, 2002). The main characteristic of e-learning is that it provides easy and universal access to learning materials and resources (Carliner, 1999; Moallem, 2003).
The Internet
The Internet fosters and supports all of the interactions of an educational experience (Ally, 2004), including the information system, with its ability to link all Internet users to information and to other Internet users (Haughey & Anderson, 1998). The Internet is a tool consisting of an information system, a communication system, a collaboration system, and an educational authoring system (Collis & Moonen, 2001). The Internet also links participants with information and with each other (Haughey & Anderson, 1998). The Internet offers a way to gain the users’ attention and “presenting opportunities for focusing perceptions and prompting recall” (Fahy, 2004, p. 165). Frequent, effective engagement in the educational information system for learning can be supported by feedback from the machine. The feedback needs to represent positive supports, high expectation, and guidelines to make things visible.
Megauniversities
The use of the Internet, particularly the Web, provides postsecondary institutions with universal educational access and flexibility (Collis & Moonen, 2001). Examples are the advent of the megauniversity and universal design for learning. Megauniversities are being established to overcome time and special physical limitations. A megauniversity is an institution with 100,000 or more students enrolled, employing technology to deliver courses. This also makes the courses cost-effective (Daniel, 1996; Hughes, 2004). The digital library supports physical flexibility of the megauniversity.
The three primary megauniversities are located in India (Gandhi National Open University, with an enrollment of 1,300,000); the United Kingdom (Open University, with an enrollment of 150,000 in over 20 countries); and the United States (University of Phoenix, with an enrollment of 230,000) (Mediawiki, 2005). These examples demonstrate that digital library users are “of all races, both genders, and all ages” (Moore, 1998, p. 2). Additionally, the users come from different cultural and linguistic backgrounds from all over the world. The large number of the digital library users also demonstrates that technology, as the learning medium, can be universally beneficial to all readers and researchers especially in light of cost and physical limitations (Smith & Meyen, 2003) by offering more flexibility (Ngwenya, Annand, & Wang, 2004). Collis and Moonen (2001) asserted that flexibility is the most beneficial function that the Internet has provided. The flexibilities are related to (a) physicalness—time and space, (b) various menus of information; topics and sequences of the different information, (c) resources, (d) delivery and logistics. As to the flexibility of the resources, the discussion included resources of (a) social organization of informatics, (b) language to be used for interaction within the research, and (c) organization of representation of information. The items discussed in relation to the flexibility of delivery were (a) when and where contact with digital librarians and other users interested in the same research topic, (b) method and technology for obtaining support and making contact, (c) types of help, communication available, and technology required, (d) location and technology for participating in various aspects of the course, and delivery channels for information, content, and communication (Collis & Moonen, 2001). The Internet also links participants with information and with the other information system users (Haughey & Anderson, 1998).
Online Learning
Postsecondary institutions of higher education responded to the instructional opportunities afforded by the Web and the Internet through the offering of courses and access of educational information system (Guri-Rosenblit, 1999). Additionally, there has been a movement to establish new universities that primarily focus on distance education utilizing the educational information system and the Internet as primary sources for the delivery of instruction (Tait, 2003). Leh (2002) stated that “It is estimated that in 2002 about 85% of two and four-year colleges would offer distance education programs and that by the year 2006 enrollment in distance education learning programs will increase by 1.5 million students” (p. 88).
This growth has been largely due to the ease of accessing instruction offered over the Internet and in combination with other forms of digital technology, such as the educational information system (Belanger & Jordan, 2000), including the digital library. Students who previously were not able to participate in higher education through traditional campus-based environments are now able to enroll and complete coursework or degree programs without relocating or changing their lifestyles (Rossman, 1992). Asynchronous learning networks and the information system allow the users to have more opportunities to access and process information, independently (Gilbert & Driscoll, 2002).
The number of postsecondary educational institutions implementing online learning continues to increase. With the increases, concerns regarding the quality of online courses and th manner in which to provide an information system, such as the digital library arose. A major concern was the need for evaluation strategies that assess the effectiveness of the information system and its design. These concerns have centered on the importance of knowing what educational information system designs maximize the user’s satisfaction, how best to structure the content of information for the digital delivery, and how to evaluate the quality of information that is valid and reliable while also providing evidence on how to improve the quality of the information system. Given the growth in virtual schools and the varied circumstances under which the information system including the digital library is offered, evaluation instruments and processes of the educational information system are needed to ensure high quality information system.
The significant issue relates to the newness of digital library as a form of the information system and the lack of evaluation and validated instruments for assessing the effectiveness of educational information system designs, which are essential to enhance the users’ usability and ensure quality digital library. In addition, digital libraries have evolved without the benefits of a knowledge base evaluating this form.
As technology has improved, the focus has been on creating online courses and on developing the capacity to employ more advanced features in offering online instruction. However, less attention has been given to empirical data determining the factors that contribute to effective educational information system designs that maximize the users’ knowledge construction. Little quantitative research has been conducted to develop any criteria or standards to evaluate the information system. The next section discusses standards in the context of the information system.
The Educational Information System
The educational information system should consist of collective and systematic learning resources to implement the e-learning experience, and facilitate the quality e-learning to gain desired information construction within a learner. It is assumed to be a part of the educational system within the academic offerings. The educational information system is standardized and evaluated by accreditation criteria that assesses all the integral quality of the knowledge building experience.
Guiding Principles for Distance Learning and Teaching was published in 1999 by the American Distance Education Consortium (1999) and was revised to four categories with 24 principles (American Distance Education Consortium, 2002). Based on the categories, the authors believe that the educational information system should be (a) a facilitator of constructive communications between information and the user, (b) user centered, (c) directed to the educational systems in the digital society where the information is the knowledge people work with, (d) an interchange between an individual’s knowledge construction and experience with others, (e) a collaborative knowledge building community, and (f) a facilitator of authentic research (American Distance Education Consortium, 1999). The e-learning experience should ensure the quality of the educational information system. American Distance Education Consortium (2002) guiding principles have been adapted for the quality information system:
The research experience of the educational information system must have a clear purpose with tightly focused outcomes and objectives.
The user is actively engaged.
The research environment of the educational information system makes appropriate use of a variety of media.
The research environments of the information system must include problem-based as well as knowledge-based research.
The research experiences of the information system should support interaction and the development of communities of interest.
The practice of research in the information system contributes to the larger social mission of education and training in a democratic society (p. 1).
In addition to these essential criteria, the educational information system must (a) design for active and effective research, (b) support the needs of users, (c) develop and maintain the technological and human infrastructure, and (d) sustain administrative and organizational commitment (American Distance Education Consortium, 2002, p. 1).
Identifying and Validating Indicators
The author’s first goal was to identify and validate critical components of a digital educational environment. The validation study warranted consideration in the evaluation of digital educational environment including web development and digital information system. The study involved:
An extensive review of the literature to identify valid indicators of digital educational features.
Recruiting subject matter experts (SMEs) to validate the indicators derived from a synthesis of the literature.
The development of an instrument designed to determine the proximity between indicators based on a pair-wise comparison of the indicators through a multidimensional scaling model.
A panel of SMEs in rating each indicator in comparison to all other indicators on the dimensions of similarity and dissimilarity.
This study identified and validated 99 indicators applicable to the evaluation of the digital educational environment at the postsecondary level. The indicators were derived from an extensive review of the literature and input from two SMEs. The SMEs independently rated the similarity and dissimilarity of indicators by making pair-wise comparisons of the indicators utilizing multidimensional scaling (MDS) techniques. The research indicated that the digital educational environment had three dimensions: each dimension was labeled as Accessibility, Adaptability, and Clarity of Communication. The study resulted in four distinctive clusters of indicators in each dimension. The four clusters are the same for each dimension, but the coordinates of the indicators within each cluster vary slightly. The clusters are: contextual accommodation, instructional access, guided learning, and organizational clarity.
Some indictors applicable to the educational information system are listed below. All the indicators were validated by the subject matter experts and are valid items in the criteria.
So that I could work at a pace that supported my needs and abilities.
To be flexible.
To be accessible to users with a wide range of technical skills and abilities.
To work efficiently in a variety of settings (home, school, or on the road).
To allow the users to work independently
To be accessible any time and any place.
To include an overall design that appropriately identifies tasks and conceptual ideas.
With a consistent user interface.
To include appropriate media for the research objectives.
With a stable and easily accessed technical delivery system.
In a manner that divides information into manageable chunks and avoids excessive scrolling.
To have aesthetically appealing screens.
To use logical layout, spacing, and density that contribute to readability.
With appropriate fonts, text size, and emphasis so that the text is easy to read and understand.
With visual and auditory options (e. g., selectable font size and audio redundancy) for students with visual and hearing impairments.
To include effective and sufficient menus.
So that I could easily find what I needed.
To use navigation labels that are clear and meaningful.
Allowing clear options to escape from a mode or page and return to a “home” or central navigation page.
To aid users in knowing where they are and where they have been within the information.
With visual and/or auditory signal that indicate the start and end of a task.
To provide explanations for how students access support services.
With features that are intuitive and require minimal technical support.
To include effective embedded help online.
To include easily accessible “on-call” support.
The users can easily find technical support when needed.
To enhance the management of all information.
To monitor all linked URLs to be certain they are always accessible.
To maintain archives of the digital library for the course.
To ease access to the information by users.
To provide students benchmarks for completing course requirements on time.
To minimize discrimination based on diverse user characteristics.
To accommodate cultural differences between the user and the information system.
To ease access to any part of the educational information system.
To accommodate users with disabilities.
To accommodate personal research preferences of users.
To allow users to vary font size.
To clearly present graphics and illustrations.
To allow users to easily communicate with other users interested in the same topic.
To provide access to the information system maps.
To promote self motivated and self directed research.
To use an effective structure for digital delivery.
Using information that is current.
Using accurate and relevant information.
Using information that actively engages users in research.
With concept maps that uses clear and understandable language.
With information that adequately covers the critical topics for the subject.
To include links to meaningful Internet outside resources that are appropriate to research objectives.
With online discussion opportunities related to the information.
The indicators can be used as the criteria of the evaluation of the educational information system, when the user interface, navigation, universal design for the human-computer interaction, and the quality of information are measured.
Three Dimensions
The “accessibility” dimension includes indicators that emphasize attributes that enhance access to the information system and to information. Navigation options, required technical skills, control of font size, and related screen management capabilities are central to this dimension. The accessibility dimension represents the cognitive domain of the educational information system. It mainly addresses the user interface, using user-friendly or human-centered strategies.
The “adaptability” dimension includes indicators that allow the user to make modifications that meet their instructional preferences. These include indicators that give the user more control over the presentation features, more choices on how information is managed, and indicators that offer more options for understanding the information and organization of the educational information system. Like the accessibility dimension, the adaptability dimension also represents cognitive domain of the structures as well as affective domain.
The clarity of the “communication” dimension includes the relevance and clarity of the educational information system, which includes the use of an explicit concept map. The dimension indicates the educational information system as the means of the effective conveyance of information. It addresses how well: (a) the information is presented, (b) the information is delivered, and (c) the learning expectations are conveyed.
The effective educational information system should have balanced cognitive, affective, and social domains. Accessibility, adaptability, and clarity of communication are three criteria of the information system. The accessibility dimension is related to Web accessibility, which means all the users can access the information without heavier workload of mind by using cognition and memory strategies thorough the structured computer human interface. The adaptability dimension can reduce human errors by optimizing users’ preferences of color, font size, or context, while interacting with the technology or information. The clarity of communication domain can be enhanced by providing feedback, conceptual maps, and any other devices to make things visible.
These criteria can be implemented into the educational information system supported by social constructivism. Social constructivists emphasized user interactivity through interaction between the system users and information and between users and users. Social constructivist approaches can include “reciprocal teaching, peer collaboration, and cognitive apprenticeships” (Schunk, 2004, p. 298). Schweizer, Whipp, and Hayslett (2002) chose social constructivist theory as a theoretical framework for digital educational system design and delivery. The positive results of the applications of social constructivist theory are: (a) incorporation of articles, Web sites, videos, audio views, and multimedia programs, (b) incorporation of guest lectures and textbooks, (c) discussion device built in the educational information system, (d) development of question prompts and discussion rubrics built in the information system, and (e) survey on the challenges and the supports the users are in need of. The applications results in a complex environment and authentic research, a knowledge community, the users’ active and collective meaning building, and support for the users.
Cognition and the Educational Information System
Spiro, Coulson, and Anderson (1988) defined Cognitive Flexibility Hypertexts as “the systematic development of nonlinear and multidimensional” digitalized educational environments (p. 22). Mehall (2002) argued that Cognitive Flexibility Hypertext was implemented in order to apply cognitive flexibility theory to the development of educational hypertext. The primary goal of cognitive flexibility theory is to select the use of knowledge to adaptively fit the needs of essential subject matter in different situations and to change one’s viewpoint when knowledge is constructed by an individual (Spiro et al., 1988; Spiro, Feltovich, Jacobson, & Coulson, 1991; Spiro & Jehng, 1990).
Cognitive flexibility theory facilitates hypermedia-learning environments to have the maximum possibility of multiple representations and avoidance oversimplification and over-regularization (Spiro et al., 1988; Spiro et al., 1991; Spiro & Jehng, 1990). Spiro and colleageus (1991) argued that users of the educational information system should develop the skills of flexible cognitive processing and be able to acquire content knowledge structures. According to Spiro and colleagues (1991), the information users are expected to be capable of restructuring their knowledge with adaptive responses to the different situations by contriving structural constituent as the best fit for the particular situation. In addition, Graddy (2001) explained that information users must experience multifaceted cases as much as they can, which could provide ill-structured, context-dependent, and complex-learning environments.
The limitations of evaluation in the digital environment are due to the fact that the digital library is a currently emerging educational mode. Based on the literature reviews, the current issues of the evaluation in this field indicate that:
Little research was done to provide authentic guidelines (McLoughlin & Luca, 2001);
Specific standards were not provided for the evaluation of the educational information system. Accreditation at any level was to legitimize the quality of digital environment yet the existing accreditation standards fit traditional educational information system (Onay, 2002).
Categories of the items to evaluate the digital library were limited (Wagner, 2001).
The evaluation was implemented at the experimental, individual level (Rovai, 2003).
The terms used for the digital library were not defined distinctively (Belanger & Jordan, 2000), and random sampling was difficult as far as research was concerned in the digital context (Gunawardena, 2001).
McLoughlin and Luca (2001) reported that “little academic research has been done to provide guidelines to implement educative, authentic or valid assessment processes” (p. 3), which fits the educational information system and digital environments. Wagner (2001) pointed out that the main problem of those measurements was that a theoretical or conceptual framework was not included in the research, so the researchers built their study on the existing work. Additionally, Wagner also asserted that the users’ attitude, and their satisfaction toward the information system, were usually assessed to evaluate the effectiveness of the educational information system.
Conclusion
The effective educational information system can be built when the three criteria of accessibility, adaptability, and clarity of communication are implemented into the design of the educational information system. Such a system should also include (a) built-in flexibility and (b) universal design for the architecture, using the user-centered psychological principles developed and revised by the American Psychological Association (APA) Work Group of the Board of Educational Affairs (APA Task Force on Psychology in Education, 1993; APA Work Group of the Board of Educational Affairs, 1997). The seven principles of universal design to facilitate of the information system design are:
Equitable use for every individual who has different cultural, linguistic background, has/has not disabilities, and different abilities;
Built-in flexibility for the user’s preference and abilities;
Simple, consistent and intuitive;
Perceptible information in the natural context using common sense;
Error tolerance reducing the errors and safe to use;
Low physical and cognitive efforts; and
Flexible physicalness—appropriate size and space for all approaches.
When the seven principles are employed into the design of the educational information system design, along with the accessibility, adaptability, and clarity of communication, the information system is highly structured.
In turn, the highly structured system allows users to exert more responsibility, autonomy, and inputs in order to gain productivity and vice versa (Moore, 1991, 1993; Moore & Kearsley, 1996). Kanuka, Collett, and Caswell (2002) asserted that high structure increased opportunity for quality communication between the information users and information; provided schema that met the users’ needs; and improved users’ research capability to exert their own autonomy over their research experience. Therefore, the successful information system was determined by the optimal amount of opportunity for the quality accessibility; appropriate adaptability which provided well-organized affective structure; and clarity of communication, which motivated the users’ autonomy in the information structure.
The main role of the educational information system evaluation should be to identify the essential points of the information system—the validity and reliability of the information, and examine and improve the quality and effectiveness of the information and educational information system. Educational information system evaluation should be a device to ensure that the system is efficient in producing the educational research to review expected outcomes.
Thus, focused empirical research on the validity and reliability of the quality of information and a measurement instrument of the educational information system will provide additional evidence of the effectiveness of these systems. Such studies will contribute to empirical evidence and theoretical rationales to support the adequacy and appropriateness of the educational information system.
