A Comprehensive Model for Evaluating E-Learning Systems Success

Education is one of the fields that has improved rapidly as a direct result for the development of information and communications technology (ICT), and stimulated to adopt e-learning. E-learning directly resulted from the integration of education and technology and is increasingly considered a powerful medium for learning.

E-learning has facilitated learning by delivering a learner-centered and interactive learning environment to anyone, anywhere, and anytime (Khan, 2005). In addition, it plays a significant role in shifting from teacher-centered to student-centered education (Taha, 2014, p. 2).

Despite e-learning’s successful implementation, a considerable number of e-learning projects fail to achieve their goals, and face slow progress and increasing dropout rate (Frimpon, 2012; Liaw, 2008). In addition, evaluating the success of e-learning systems is still an issue facing e-learning stakeholders.

ROA significant number of studies have focused on the issue of e-learning success. In fact, they fulfil the needs of e-learning stakeholders to a certain extent but do not meet all of the requirements. There remain disagreements about the factors that are most influential in measuring e-learning systems’ success. This direction of research has received little attention for developing an overarching model that can assess e-learning systems’ success from different perspectives.

This study aims to fill this void by proposing a comprehensive model for evaluating the success of e-learning. The significance of this study is in identifying the determinant factors and constructs impacting the success of e-learning systems and group these factors in a model that is believed to be holistic because different perspectives are considered in developing the model.

E-learning systems are multidisciplinary systems, for which consensus on their definition has not been achieved (Al Sabaway, 2011; Lee, Choi, & Kang, 2009; Ozkan & Koseler, 2009). Many researchers have evaluated e-learning systems from computer science, information systems, psychology, pedagogy, and technology perspectives.

Various methods, frameworks, and models have been introduced to measure e-learning systems’ success. The contributions to evaluating e-learning systems’ success can be categorized into four such approaches: technology acceptance model (TAM); the DeLone and McLean IS success model (D&M); user satisfaction models; and e-learning quality models.

TAM is a widely used model in the information system field. It was developed first in 1989 (Figure 1) to measure the success of a new technology in terms of the acceptance and use of this technology. The model presumes that there are factors that impact the users’ decisions when they face a new technology.

In the context of e-learning, many studies adopted TAM to evaluate the success of e-learning in the same manner as information systems success (Hayashi, Chen, Ryan, & Wu, 2004; Lee, Choi, et al., 2009; Liaw, 2001; Limayem & Cheung, 2008; Martins & Kellermanns, 2004; McFarland, 2001; Ngai, Poon, & Chan, 2007; Ong & Lai, 2006; Roca, Chiu, & Martínez, 2006; Sánchez & Hueros, 2010; Selim, 2003, 2007; Stoel & Lee, 2003; Wang & Chiu, 2011; Yi & Hwang, 2003). These studies vary between validating and testing the robustness of the model by providing empirical evidence on the existing relationships between model factors, to studies that have changed the model’s constructs and extended it to include factors applicable in the context of e-learning.

From the studies found in the literature, it is evident that TAM is a commonly used model. TAM has been adopted and/or extended to include other factors that influence the acceptance of e-learning. It is concluded from previous research that some factors are more outstanding than others and have a significant impact on the acceptance of e-learning systems, for example, self-efficacy, while others need more investigation, for instance, habit, gender, and perceived resources.

The D&M model is a common model that has been used for measuring the success of information systems. It has been extensively cited in academic papers, and reportedly used in over 300 (Delone & McLean, 2003). The D&M model is one of the most important models in information systems. It first appeared in 1992 and was updated in 2003 to include six constructs (Figure 2): system quality, information quality, service quality, use, user satisfaction, and net benefit.

The D&M model was applicable in the field of e-learning in the same manner (Adeyinka & Mutula, 2010; Almarashded, Noraidah, Azan, & Mukhtar, 2010; Hassanzadeh, Kanaani, & Elahi, 2012; Holsapple & Lee-Post, 2006; Hsieh & Cho, 2011; Klobas & McGill, 2010; Lee & Lee, 2008; Lin & Lee, 2006; Lin, 2007, 2008; Masrek, Jamaludin, & Mukhtar, 2010; Wang & Wang, 2009). The validity of the model has been tested by measuring the success of e-learning as a whole or partially and others have extended this model by including other factors that influence the success of e-learning. Other researchers have combined the model with other models and theories to explore widely the factors affecting the success of e-learning systems (Al Sabawy et al., 2011).

The D&M model has been successfully used for measuring the success of different e-learning systems and most of the studies empirically demonstrated its validity and reliability.

The user satisfaction approach has been used widely by researchers in the field of e-learning (Kang & Lee, 2010; Leclercq, 2007; Ong & Lai, 2007; Pike Tayles & Abu Mansor, 2010). Sun, Tsai, Finger, Chen, & Yeh (2008) developed a six-construct model to measure e-learning based on learner, instructor, course, technology, design, and environment (Figure 3). The results of the study revealed that learner computer anxiety, instructor attitude toward e-learning, e-learning course flexibility, e-learning course quality, perceived usefulness, perceived ease of use, and diversity in assessments are the critical factors affecting learners’ perceived satisfaction (Sun et al., 2008).

Ozkan and Koseler (2009) assessed the user’s satisfaction with learning management system (LMS) and proposed a multidimensional model via six dimensions (Figure 4): system quality, information quality, service quality, supportive factors, learner perspective, and instructor attitudes.

In 2010, Naveh, Tubin, and Pliskin conducted a study to investigate students’ use and satisfaction of LMS and the relation between these two factors and organizational factors. According to this study, use and satisfaction are significantly correlated with organizational variables: course content and size, instructor status, and the existence of interactive functionalities like forum showed significant correlation with LMS use. The study also reveals low correlation between course discipline and satisfaction.

Different approaches and models have emerged to assess the overall quality of e-learning, for example, MacDonald, Stodel, Farres, Breithaupt, & Gabriel’s (2001) demand-driven learning model (DDLM) (Figure 5).

The demand-driven learning model was developed to evaluate the benefits of web-based learning. It has five main components: the quality standard of “superior structure,” three consumer demands (content, delivery, and service), and learner outcomes (MacDonald et al., 2001).

Another approach to measure the quality of e-learning was introduced by Ehler (2004) based on the learner’s perspective. This study was developed to identify the critical indicators adopted by learners to evaluate the quality of e-learning. The study identified seven main constructs used by learners for assessing the quality of e-learning: tutor support, cooperation and communication in the course, technology, cost-expectations-value, information transparency, course structure, and didactics.

In 2007, Ehlers provide a new model for e-learning quality based on the participation of educational stakeholders. The model identified four dimensions of quality: knowledge, experience, innovation, and analysis.

Pawlowski, Barker, and Okamoto (2007) presented ISO/IEC 19796-1 and compared it with existing approaches of e-learning quality. Abdellatief, Sultan, Jabar, and Abdullah (2011) proposed a model for e-learning quality based on developer’s view with four main measurement indicators service content, system functionality, information technology and system reliability and proposed 11 subcharacteristics with its attributes by following the structure of standard IOS/IEC 912.

Considerable research has focused on measuring e-learning quality and proposing models and indicators for this purpose. However, the complexity and generality of the “quality” concept is an issue encountered by researchers. In addition, the varied e-learning stakeholders put more pressure to identify the measurements most suitable for each group.

In order to develop a model for evaluating the success of e-learning systems, we used the constructs of the four models: (D&M) model; TAM; User Satisfaction Models; and E-learning Quality Models. In spite of all their strengths, the four models still have defects (Hassanzadeh et al., 2012), they partially fulfil the needs of e-learning success assessment, and they need to be customized for e-learning areas. In addition, there is still room for improvements and extensions (Marangunic & Granic, 2013). In this article, to propose a more comprehensive model, a synthesis of these previous models is presented.

Based on the results and analysis of the literature review, a conceptual model is proposed. The model is believed to be comprehensive for measuring the success of e-learning system. The model is based on six perspectives: quality; usefulness; satisfaction; user attitude; social factors; and benefits of using the system.

The selection of the model constructs is based on the four approaches for evaluating the success of e-learning: DeLone and McLean, TAM, Satisfaction, and Quality approaches and based on their importance in measuring the success of e-learning systems field.

In order to build a model for evaluating e-learning systems success we used, first, the D&M model measurements. Because this model was developed to measure information systems success and these systems have no pedagogy theme, it has to be customized for the e-learning area.

The technical quality is another important determinant of the quality of e-learning, and technical problems strongly influence the overall success and satisfaction of users. On the other hand, students are very concerned about the quality of the information (course content) to be clear, easily understandable, appropriate breadth, and has up-to-date content. As a result, a more customized version, to meet the specific needs of the students, is needed.

So we incorporated the Quality construct with four measures (Technical System Quality, Pedagogical System Quality, Information Content, and Service Quality). The System Quality was decomposed into two factors, technical and pedagogical system quality, as suggested by Hassanzadeh et al. (2012) and Ozkan and Koseler (2009) to be appropriate in the context of e-learning.

Technical System Quality is related to technical success of the issues related to the system (DeLone & McLean, 2003) and measured by the indicators in Table 1.

Pedagogical System Quality consists of the quality measures according to the educational functionalities and capabilities that facilitate teaching and learning (Lee 2010; Hassanzadeh et al., 2012), for example, existence of features like chats and forums that facilitate interactivity and communication with other students and instructors. The Table 2 summarized the pedagogical system quality factors.

Information Quality (Content) is the measure of system semantic success (Delone & McLean, 2003) that is related to the quality of the output (Wang & Wang, 2009) (see Table 3).

The last theme in the Quality construct is Service Quality. The quality of the service delivered through electronic media has received noteworthy attention in the context of e-learning (Al Sabawy, 2012). Four indicators were employed to gauge service delivery quality in e-learning (see Table 4).

Benefits of using the system, in a restricted sense, is the impact of using the e-learning system on an individual or group. This construct assesses the different benefits obtained from using the system. In a broader sense it is the benefit to the organization and community as a whole. For the purposes of developing our model, only benefits of using the system on individuals are considered with three determinants: achieving goals; system loyalty; learning benefits. The broader benefits of using the e-learning systems are beyond the scope of the present study, so it was excluded. Indicators of the benefits construct supported by related studies are presented.

Achieving Goals is one of the components that has a significant role in measuring the success of e-learning and has to be included in our model. It measures the acquisition of skills that influence achieving the personal goals and improving the academic development of students (see Table 5).

System Loyalty is another factor included in our model, which is related to students’ involvement and dependence on the e-learning systems (see Table 6) (Hassanzadeh et al., 2012; Lin & Lee, 2006).

Learning Benefits are used to measure the student’s performance improvement resulting from using the e-learning system and other benefits of learning in terms of saving students’ time in searching for the information and course materials (see Table 7).

In respect to TAM, ease of use, perceived usefulness, and use are considered the major constructs in this model. The evidence presented by previous studies support the selection of the three constructs to measure e-learning system success. Consequently, it was included in this model.

Ease of Use was defined, according to Davis (1989), as “the degree to which an individual perceives using the e-learning system free of effort” (p. 319). In the e-learning era prior researchers adopted “ease of use” as a central determinate of student satisfaction and the success of e-learning systems. Indicators for ease of use are shown in Table 8.

Perceived Usefulness is a construct employed in this model to predict different factors. It was defined as “The degree to which a person believes that using a particular system would enhance his or her job performance” (Davis, 1989, p. 319). Empirical research has showed the reliability of this construct as a predictor of intention to use. Davis (1989), Joo, Lim, and Kim. (2011), Drennan, Kennedy, and Pisarski (2005), and Hsieh and Cho (2011) found that perceived usefulness had the strongest effect on student satisfaction among the factors that predicted this construct. Table 9 shows the determinants used to gauge this construct.

Intention to Use is an attitude toward using the system (Davis, 1989; DeLone & McLean, 2003) and is defined as the users’ decision to use the system before actually doing so (Hassanzadeh et al., 2012). Table 10 lists details of some studies that targeted identifying the intention to use in the context of e-learning.

User Satisfaction is a fundamental measurement in the success and acceptance of technology. Several studies considered satisfaction as a single construct to evaluate the success of an e-learning system (DeLone & McLean, 2003) or as multiple constructs; (Sun et al., 2008; Ozkan & Koseler, 2009). It was found that user satisfaction is a valuable learner’s attitude construct to incorporate in our model that was validated and supported by several studies (see Table 11).

Social Factors have been considered an important construct in measuring the success of e-learning. Ozkan and Koseler (2009) considered e-learning systems as a sociotechnical entities and the success of e-learning as a combination of “social issues” and “technical issues” and other circumstances. Previous research supported e-learning as social entity being an important indicator for successful systems (Liaw et al., 2007; Selim, 2007; Wang et al., 2007). The technical part in our model is covered in the quality construct adopted from the DeLone and McLean model. Accordingly, social factors with three major determinants (learners, instructors, and supportive issues) are added to our model (see Tables 12 and 13).

As mentioned earlier in this article, quality of e-learning is a complicated concept and metrics for measuring the quality of e-learning are diverse based on different perspectives of different stakeholders. A significant contribution to measure the quality of e-learning, which has been presented by several researchers and has been tested and confirmed in studies, are the supportive issues which are incorporated in our model as “support factors” under social factors construct based on the model proposed by Ozkan and Koseler (2009) (see Table 14).

Another important factor presented under the fourth approach is Academic Performance, which was employed in the Lee and Lee (2008) model. Academic performance is included in our model under the Benefits as suggested by (Hassanzadeh et al., 2012; Lee & Lee, 2008).

According to previous studies on e-learning and the performed analysis, the constructs, factors, and relations between model constructs were identified. As a result, a model for evaluating the e-learning systems success EESS is presented (Figure 6).

This study has proposed a model for evaluating e-learning systems success (EESS) encompassing a collective set of measures associated with e-learning systems. In this article, we have proposed a comprehensive model based on different perspectives in relation to quality, usefulness, social factors, user satisfaction, attitude, and benefits of using the e-learning systems. However, several researchers confirmed, “the success of e-learning is a multidimensional and interdependent construct” (Delone & McLean, 2003) and it is essential to examine the interrelationships between these constructs (Hassanzadeh et al., 2012; Ozkan & Koseler, 2009) in more detail. Therefore, future research efforts will be conducted to focus on and test the relationships between the proposed model constructs within the context of e-learning. Another future endeavor will be to check the validity of the model on learning management systems.

The EESS model is not a fixed and unchanged model and is subject to further and continuous developments. Future research may extend this model through adding the organizational perspective and other indicators to cope with the continuous development and changes in the e-learning field. In this regards, the EESS model is composed of the major constructs and factors which are basics for successful evaluation of e-learning. In conclusion, 52 measures grouped under 7 constructs for measuring the success of e-learning systems will be of great benefit to those involved in e-learning as a guidance to gain a better understanding of the issues related to evaluating the success of e-learning systems.