Research on Internet-Supported Learning: A Review

Research on learning supported by technology is as varied as it is prolific. Nearly as many studies have concluded that technology-supported learning makes no significant difference as those that have found it does. For example, some research on learning supported by such technologies as e-mail, the Web, and course management systems (Clark & Mayer, 2003; Keating & Hargital, 1999) concluded that relying on Internet-Supported Learning (ISL) could at times be more effective than its traditional, classroom counterpart (Abel, 2005; Allen & Seaman, 2004; Black, 2002; Faugli, 2003; Gilliver, Randall, & Pok, 1998; Jewett, 1998; Koory, 2003; Ladyshewsky, 2004; Motteram, 2006; Ngu, 2002; Reasons, Valadares, & Slavkin, 2005; Schutte, 1996; Torre, Pfeifer, Lamb, Walters, Sebastian, & Simpsonet, 2004). Conversely, other research found no significant differences (NSD) when comparing ISL environments (Arvan, Ory, Bullock, Burnaska, & Hansonet, 1998; Broad, Matthews, & McDonald, 2004; Davis & Graff, 2005; Frederickson, Pickett, Shea, Pelz, & Swan, 2000; Hammonds, 2003; Hiltz, 1997; Hiltz, Coppola, Rotter, Turoff, & Benbunan-Fich, 2000; Lim, Morris, & Kupritz, 2006; Ostiguy & Haffer, 2001; Parker & Germino, 2000; Shapley, 2000; Sinyor, 1998; Wegner, Holloway, & Garton, 1999; Wegner, Ken, & Allan, 1997; Witta, 2005; Yuan, Gay, & Hembrooke, 2006). Reviewed independently, these studies might have led researchers to presume the overall effectiveness of ISL was still inconclusive. However, additional insight could be gained by comparing commonalities and differences of all such studies.

Some research had already considered the importance of comparing multiple studies in such a fashion. However, such research either reviewed only a limited number of overall studies (Abel, 2005; Allen & Seaman, 2004; 2005; Reasons et al., 2005) or had become outdated (Clark, 1985; Waxman, Lin, & Michko, 2003). Because such research did not look at large numbers of studies or was outdated, this review of studies was conducted. By using a comparative, qualitative analysis methodology (Maykut & Moorehouse, 1994), the authors analyzed significant factors embedded in 29 ISL-related research articles, many of which themselves reviewed multiple studies. The authors identified specific issues reported both in studies that indicated measurable positive conclusions (not necessarily defined solely through grades or exam scores) as well as studies that indicated NSD conclusions. Issues identified in this study can then better inform instructional design, practice, and further research. This study has the following major sections: review purpose, research questions, definitions, review methodology, summary of studies with positive results, summary of NSD studies, and discussion and recommendations.

To identify major issues of importance, this article reviews studies conducted between 1995 and 2006 on the effectiveness of ISL environments, which is more fully explained in the Review Methodology section below. It explores the effectiveness of ISL as revealed by such studies. It utilized a qualitative methodology to identify key trends and patterns. A further purpose was to point out methodological and theoretical issues that characterize such studies. Overall, such a holistic and critical review of literature can bring about a better understanding of contemporary research, which may in turn better inform instructional design, practice, and further research.

The research answered the following questions:

1. How did the Internet affect learning in the environments studied?

2. What methodological and theoretical issues characterized Internet-supported learning research (ISLR)?

3. What, if any, patterns and/or trends existed in these research studies?

4. What implications existed for future research studies?

Buzzwords abounded in the educational technology research literature reviewed. Electronic- or e-learning, learning technology, instructional technology, information and communications technology, educational technology, computer or Internet-based learning, Web-supported/assisted/based/mediated learning, online learning, a/synchronous learning, conventional/traditional/classroom learning, and on-campus/off-campus/blended learning were some of the most frequent terms. Some were distinct in meaning and others were alternatively used, many interchangeably. Such diverse definitions might have drawn confusion to the general readership. Consequently, this paper limited itself to only some of these terms: ISL, blended learning, online/on-campus learning, traditional/conventional/classroom learning, and educational technology. These and other terms were operationally defined as follows.

Online learning. According to Allen and Seaman (2005), an online course requires at least 80% of its content to be delivered online. In this paper, an online course describes the fact that students learn from, through, and/or with the Internet. There is no physical interaction in online learning.

Blended learning. In a blended/hybrid course, 30 to 79% of course content is delivered online and the remaining part of the course is delivered on-campus (Allen & Seaman, 2005). In this article, any type of learning that mixed online and on-campus activities on a given course, regardless of the extent of mix, was considered blended or hybrid.

ISL. ISL involves the use of the Internet for academic purposes (Abel, 2005). It includes fully online and blended learning but not e-learning, as the latter is more inclusive and includes learning that uses any electronic device. Research on ISL is called ISLR and environments supported by ISL are called ISLE.

Traditional/conventional/classroom learning. This learning is the usual way of learning within the classroom context. It may include lectures, discussions, projects, and/or assignments. It does not, however, include the use of educational technology, in its narrower sense, for academic purposes.

Educational technology. Although educational technology refers to the systematic use of any method of learning and teaching including such concrete technologies as projectors, televisions, radios, computers and the Internet (Bates & Poole, 2003; Eraut, 1996; Januszewskli, 2001; Thomas, 1987), its meaning in this article is delimited to include Internet-based technology. Any research that includes ISL is referred to as educational technology research to simplify terminologies. All of the studies referred to as educational technology research included at least one Internet-based technology. None included sole delivery by television, correspondence, or other non-Internet-based technologies.

Effectiveness. Many studies measured the effectiveness of ISL by considering student outcomes. Usually, scores and/or grades earned from tests, assignments, projects, mid-exams and final exams were used to gauge the effectiveness of ISL. However, in many cases, assessment was defined more qualitatively through final projects designed to meet course or program objectives as well as quantitatively through measures of student satisfaction. Consequently, while this article considered letter or percent grades an important indicator of effectiveness, conclusions could not be based solely on grades as an assessment methodology. A broader assessment including project-based learning objectives, field observations, measures of student satisfaction, and instructors’ anecdotal comments was considered. However, all the assessment strategies addressed student subject knowledge in one fashion or another and were measurable. In short, this review defined effectiveness as it was defined and reported in each research study. To do otherwise would have eliminated most of the studies from the qualitative review process.

This review assumed that all research types, if properly designed and executed, are valuable in the production of knowledge. Thus, no discrimination was made between the quantitative and qualitative paradigms. The review was not a meta-analysis, which makes preference to studies with focused hypotheses (Hoyle, Harris, & Judd, 2002). Although meta-analysis is more powerful in reviewing quantitative studies based on some carefully selected criteria, it cannot sufficiently represent all ISLR. Meta-analysis would have been confined to quantitative data. Thus, examining the effectiveness of ISL environments, revealing methodological and theoretical issues in research, and discovering patterns and trends in ISLR based solely on an analysis of quantitatively driven studies would have been impractical for several reasons. First, there are several qualitatively-driven studies that examined the topic under discussion. In such cases, “a narrative literature review might be the best one can do” (Hoyle et al., 2002, p. 491). Second, even quantitative studies did not formulate and test similar or at least focused hypotheses, a requirement in meta-analysis. Third, the number of studies on the effectiveness of ISL environments at the higher education level is very limited. Selecting quantitative studies only from this limited stock of literature for meta-analysis is itself limiting. Thus, a qualitative review was found to be more practical in revealing key issues embedded both in qualitative and quantitative studies.

In identifying and selecting the studies for the review, four strict criteria were employed. First, studies were conducted on higher education institutions such as colleges, university colleges, universities, and/or other tertiary-level institutions. Second, studies examined Internet-based, learning technologies. Technologies studied may have been asynchronous and/or synchronous, multimedia and/or text only, intranet and/or global network, or a combination of some or all of these. The World Wide Web (Web) and e-mail were the most common forms of technologies used to support learning. The third criterion was that studies examined the effectiveness of ISE environments in some manner. Outcome measures might have included grades, exam scores, objectives met, motivation, satisfaction, support, or participation. A final criterion was that studies were published between 1995 and 2006. This 12-year period was selected: (a)because studies before 1995 were not as relevant to modern implementations of ISL; (b)since this review was initiated at the beginning of 2007 and utilized a qualitative analysis method requiring significant time for transcription, coding, and analysis, it would have proven tedious to incorporate newer studies as they were published; and (c) because many previous reviews limited analysis between 2 and 5 years, an extended timeframe was desired.

In order to ensure an exhaustive representation of appropriate studies would be included in this study, in addition to the above criteria, the authors incorporated the following into the literature review process:

1. A wide range of literature resources were consulted including but not limited to: EBSCO Host, ERIC, FirstSearch, Academic Search Premier, educational technology references such as indexes and encyclopedias, Google Scholar, EdResearch On-line, ProQuest dissertations and theses, Educational & Information Technology Library (Ed/ITLib AACE), and of course searching the stacks.

2. Articles were analyzed only when from major journals, authoritative books, Web sites, from well-known research-based organizations, and/or included in educational technology references, indexes, abstracts, or encyclopedias.

3. Both authors conducted searches separately and then compared them to ensure the greatest numbers of sources were found.

As far as evidence for credibility was gained, any study was included for review regardless of research design employed, courses studied, or the nature of results reported. The authors assumed that if research already fulfilled item 2 above, it was necessarily worthy of inclusion. Through this process, a total of 29 studies satisfied all the criteria set to represent contemporary research on ISL environments.

In order to better understand what specific issues were reported in these studies, coding of data was conducted by hand using the constant comparative method (Maykut & Morehouse, 1994). The researchers collaborated in the coding process and data analysis to ensure interrater reliability, or “agreement among raters” (Vogt, 1993, p. 114). According to the procedure outlined by Maykut and Morehouse (1994), the constant comparative coding proceeded in the fashion described below.

First, studies were divided among the researchers. Researchers then conducted independent coding of studies to which they were assigned. To accomplish this, the researchers independently organized the data according to “chunks” into a table (a chunk of information was determined thematically). The data were coded using inductive category coding and simultaneous comparing of units of meaning across categories in which each “chunk” was given a code to indicate its “theme” and then each subsequent coded “chunk” was compared to previous chunks to determine whether there existed a similarity to a previous “theme” so that a new “chunk” would receive a previous or be assigned a new code.

Once all “chunks” were coded, the researchers collaborated to refine themes by consolidating codes determined to be similar. Relationships and patterns were identified and themes were collapsed. Finally, data were integrated and counted, patterns analyzed, and coded themes were consolidated by category into a matrix in column format. The researchers reviewed the matrix and differences and similarities were discussed for reliability. A final, single, properly-coded matrix was completed (see Appendix).

During the review process, the authors identified 13 studies (Abel, 2005; Allen & Seaman, 2004; Black, 2002; Faugli, 2003; Gilliver et al., 1998; Jewett, 1998; Koory, 2003; Ladyshewsky, 2004; Motteram, 2006; Ngu, 2002; Reasons et al., 2005; Schutte, 1996; Torre et al., 2004) that reported measurable quantitative or qualitative positive conclusions when examining the influence of ISL environments. Such studies have proliferated over the past ten years with the popularization of ISL delivery models. Interestingly, such studies varied greatly in methodology, environments studied, and conclusions generated. As the purpose of this study was to provide a qualitative methodology for understanding possible trends emerging from the studies rather then to analyze their relative value independently, the following findings may not necessarily be generalizable overall. That is, the purpose of this particular study was less of a “how-to” integrate ISL and more of a method to understand what the extant research indicated conclusively—in short, to analyze what trends emerged from the research as a whole.

Following is a summary of basic demographic information for the 13 reviewed studies that indicated some measure of positive conclusions. Figure 1 indicated all the studies were spread over ten years relatively evenly. According to the matrix (see Appendix), just over half of the studies (54%) were specific to undergraduate education (Black, 2002; Faugli, 2003; Gilliver et al., 1998; Jewett, 1998; Koory, 2003; Ngu, 2002; Schutte, 1996) and the other half were divided evenly with 23% a combination of graduate and undergraduate (Abel, 2005; Allen & Seaman, 2004; Reasons et al., 2005) and 23% graduate education only (Ladyshewsky, 2004; Motteram, 2006; Torre et al., 2004).

In addition, about half (46%) the studies were actual comparisons of traditional face-to-face versus ISL delivery models (Black, 2002; Gilliver et al., 1998; Koory, 2003; Ladyshewsky, 2004; Schutte, 1996; Torre et al., 2004) and the other half were divided nearly equally, with 23% studying ISL only environments (Jewett, 1998; Motteram, 2006; Ngu, 2002) and 31% studying a combination of ISL and non-ISL environments not compared to traditional delivery models (Abel, 2005; Allen & Seaman, 2004; Faugli, 2003; Reasons et al., 2005). The type of ISL models common among the 13 studies were divided almost equally, where 23% were varied in their approach (Abel, 2005; Allen & Seaman, 2004; Reasons et al., 2005), 15% relied mostly on course-management systems (Koory, 2003; Ladyshewsky, 2004), 31% used some form of hybrid delivery (Black, 2002; Faugli, 2003; Gilliver et al., 1998; Ngu, 2002), and 31% relied mostly on Web-based content (Jewett, 1998; Motteram, 2006; Schutte, 1996; Torre et al., 2004). Although nearly all the studies relied on grades to measure student achievement in one form or another, specific achievement measures most relied upon to base overall conclusions included: 15% for grades (Koory, 2003; Ladyshewsky, 2004), 23% for exams not always reported as grades (Ngu, 2002; Schutte, 1996; Torre et al., 2004), 31% for objectives met (Allen & Seaman, 2004; Faugli, 2003; Gilliver et al., 1998; Jewett, 1998), 15% for student perceptions (Black, 2002; Reasons et al., 2005), and 15% for a combination of two or more of the former measures (Abel, 2005; Motteram, 2006).

In addition to basic demographic information and in accordance with the review methodology of this study, all 13 studies indicating some measure of positive conclusions were coded qualitatively and analyzed for patterns and themes. As represented by Figure 2, four significant themes emerged from this process: (a) ISL positively influenced student achievement as measured by grades, exam scores, or objectives met (Abel, 2005; Black, 2002; Faugli, 2003; Gilliver et al., 1998; Jewett, 1998; Koory, 2003; Ladyshewsky, 2004; Motteram, 2006; Ngu, 2002; Reasons et al., 2005; Schutte, 1996; Torre et al., 2004); (b) ISL positively influenced student participation (Abel, 2005; Allen & Seaman, 2004; Gilliver et al., 1998; Jewett, 1998; Koory, 2003; Motteram, 2006; Ngu, 2002; Reasons et al., 2005; Schutte, 1996); (c) successful ISL integrated appropriate support services (Abel, 2005; Allen & Seaman, 2004; Faugli, 2003; Koory, 2003; Ladyshewsky, 2004; Ngu, 2002); and (d) successful ISL enjoyed administrative involvement (Abel, 2005; Allen & Seaman, 2004; Faugli, 2003; Ladyshewsky, 2004; Motteram, 2006). Note that multiple themes emerged from most studies. In fact, in the area of student achievement, all but one study indicated ISL positively affected student achievement in some measurable fashion (92%). Most studies also indicated ISL positively affected student participation (69%). In addition, nearly half the studies indicated successful ISL had either appropriate support services (46%) and/ or administrative involvement (38%).

When analyzing the studies by date published, several differences and similarities were noted. First, more recent studies from 2003 forward (Abel, 2005; Allen & Seaman, 2004; Faugli, 2003; Ladyshewsky, 2004; Motteram, 2006) indicated a greater focus on administrative involvement for those ISL environments deemed successful. Earlier studies had little or no mention of administrative involvement. These more recent studies reported the nine following subthemes as significant with regards to administrative involvement:

• ISL was consistent with the institutional mission,

• institutions had a desire to grow student enrollment,

• there was active administrative involvement,

• the integration of ISL provided a chance for institutions address overall missions,

• ISL supported enrollment growth with no lessening in sight,

• ISL was crucial to the long-term strategy of institutions,

• institutions themselves reported ISL was as good as traditional delivery, and

• institutions desired that a strong model for ISL be documented.

Overall, these five studies (Abel, 2005; Allen & Seaman, 2004; Faugli, 2003; Ladyshewsky, 2004; Motteram, 2006) revealed the significance of some form of institutional support for successful ISL environments. This finding was significant in that what has long been an assumption, that support from the topdown makes a positive difference, could now be qualified in some manner. In short, that 38% of these studies independently concluded institutional support was important exemplifies the importance of such support for ISL.

In addition to institutional involvement, recent studies from 2002 forward indicated a greater focus on providing support services (Abel, 2005; Allen & Seaman, 2004; Faugli, 2003; Koory, 2003; Ladyshewsky, 2004; Ngu, 2002). Earlier studies mentioning support services tended to focus on instructor-side support only. These newer studies reported the following subthemes as significant for support services:

• both faculty and student helpdesks were made available,

• institutions provided significant course development assistance to faculty,

• there was one-to-one course and instructional design assistance,

• technical training was available to students and faculty,

• technical assistance was available to students,

• students were provided an “orientation” to ISL,

• students were given significant feedback about their performance in ISL settings,

• a program coordinator was available, and

• there were concrete methods for ensuring student success with technology.

Overall, these studies indicated the importance of providing appropriate support services for successful ISL. Further, the studies provided specific recommendations as to the type of support made available. Once again, the studies provided a foundation for recommending that all ISL integrate significant support services.

In addition to institutional involvement and support services, when looking at studies that reported positive conclusions, several themes also emerged from the areas of student achievement (Abel, 2005; Black, 2002; Faugli, 2003; Gilliver et al., 1998; Jewett, 1998; Koory, 2003; Ladyshewsky, 2004; Motteram, 2006; Ngu, 2002; Reasons et al., 2005; Schutte, 1996; Torre et al., 2004) and student participation (Abel, 2005; Allen & Seaman, 2004; Gilliver et al., 1998; Jewett, 1998; Koory, 2003; Motteram, 2006; Ngu, 2002; Reasons et al., 2005; Schutte, 1996). These themes emerged from both recent as well as older studies and were not limited to specific grade levels or other demographics. In short, these themes emerged from a variety of studies at a variety of levels. This points to the possibility that these patterns were common to ISL in general at many levels, at least in higher education.

There were some differences evident in the area of student achievement when analyzed by grade level. First, when studies were from varied grade levels and utilized varied ISL models (Abel, 2005; Reasons et al., 2005), the following subthemes were reported:

• a programmatic approach was better than a course by course approach;

• the best programs used a feedback and redesign process; and

• in general, completely on-line ISL indicated higher achievement than either hybrid or traditional delivery methods that showed no differences.

These studies were especially significant since they were among the most comprehensive of all the studies reviewed. The studies taken together provided a strong argument for implementing ISL programmatically through iterative redesign versus integrating ISL simply on a course-by-course basis.

Next, when studies were reviewed that focused on graduate level education supported by Course Management Systems and multimedia (Motteram, 2006; Ladyshewsky, 2004; Torre et al., 2004), the following subthemes were reported:

• ISL was better than traditional learning when comparing grades but such differences were not statistically significant,

• smaller class sizes (<20) were better for student achievement,

• ISL was better than traditional delivery with multimedia integration, and

• ISL augmented group interaction and dialog that promoted social learning.

These studies were significant for graduate programs by indicating that smaller class sizes, multimedia integration, and group interaction were important at the graduate level.

Finally, when studies focused on undergraduate education supported mainly by hybrid and/or Web-based-only ISL models (Black, 2002; Faugli, 2003; Gilliver et al., 1998; Jewett, 1998; Koory, 2003; Ngu, 2002; Schutte, 1996), the following subthemes were reported:

• ISL was better when using feedback as part of an iterative design process;

• in many cases, ISL students could do better than traditional ones;

• in most cases, ISL rewarded students adopting adult learning styles;

• more interaction with the content, peers, and instructors was possible than with lecture formats; and

• many ISL courses had higher student satisfaction when compared to traditional delivery in student perceptions of the usefulness of material, the flexibility of the learning environment, overall student achievement, and overall student satisfaction.

These studies in the undergraduate arena revealed some significant conclusions regarding ISL. First, ISL was most successful when there existed a redesign process incorporating feedback. Interestingly, the same conclusion was reached in graduate-level studies. Second, ISL in many cases could outperform traditional delivery methods with the same content. Next, ISL students needed to adopt adult learning styles, including being more self-motivated than the traditional student. Additionally, ISL provided more interaction opportunities than the traditional classroom. And finally, student perceptions and satisfaction were generally higher in ISL environments than traditional ones.

Nine studies indicated positive influences on student participation (Abel, 2005; Allen & Seaman, 2004; Gilliver et al., 1998; Jewett, 1998; Koory, 2003; Motteram, 2006; Ngu, 2002; Reasons et al., 2005; Schutte, 1996;). These studies were not limited by grade level or other demographics. The following themes were noted when analyzing these studies:

• students were satisfied with ISL overall;

• there were no differences between hybrid, fully online, and traditional environments with regard to student participation;

• ISL graduate students tended to be more satisfied than undergraduate and specialized students;

• students spent more time in lecture mode in traditional courses than in ISL;

• ISL students interacted more with instructors and peers;

• interaction in ISL discussions was the biggest measure of success;

• ISL students showed increased motivation through participation; and

• there was better access to content in ISL environments.

Overall, the studies showed that students were generally satisfied with ISL. More interestingly, these studies generally concluded that ISL students had more opportunities for collaboration and interaction with instructors and peers, interaction in discussions was important for measuring the success of ISL, and students showed increased motivation to participate in ISL.

While findings were based on grouping all positive results studies holistically by theme for analysis, outlining specific, exemplary research studies reinforced the strength of these findings. Four representative studies (Abel, 2005; Allen & Seaman, 2004; Faugli, 2003; Koory, 2003) were selected to qualify these findings.

Abel (2005) studied a variety of models including fully online programs, fully online courses, hybrid programs, hybrid courses, Web-supported courses, and emporium-style courses. The study was one of the most comprehensive analyzed and looked at student outcomes, growth in enrollments, as well as student and faculty satisfaction and retention. Regarding student achievement, the study concluded a programmatic versus a by-course approach was most successful to implementing ISL. For student satisfaction, the study found students were satisfied with online learning overall. With regard to support services, the study indicated several critical services were needed to maintain successful ISL including faculty helpdesks, technical-skills assistance to students and faculty, course and instructional design development assistance to faculty, student orientations, continuous feedback and program assessment, and an overall program coordinator. Finally, regarding administration, the study concluded successful ISL programs maintained consistency with institutional missions, intended to use ISL to increase enrollments, had active involvement from administrators, and used ISL to look at the overall mission of the institution.

Allen and Seaman (2004) also studied a variety of models including traditional courses with no ISL, Web-enhanced courses no more than 29% online, hybrid courses between 30% and 79% online, and courses considered fully online at 80% or above. The study analyzed student outcomes as well as grades. The study made no significant claims regarding student achievement. Regarding student satisfaction, the study found students in medium to large schools (over 1,500 in enrollments) were more likely to be at least as satisfied with online learning versus smaller schools. In addition, the study found doctoral, masters, and associate of arts students to be generally more satisfied than bachelors or specialized students. For support services, the study found courses with methods for ensuring student success with technology had better measurable student outcomes than those that did not. Finally, administratively, in on-line environments, enrollment growth was realized, ISL was crucial to long-term institutional strategies, and all findings supported on-line learning was as successful in terms of student outcomes as traditional approaches.

Faugli (2004) studied undergraduate students involved in hybrid courses supported by net-based learning environments (NBLE) that incorporated collaborative and problem-based learning models. The study looked at measurable student outcomes for analysis. For student achievement, the study found NBLE to more successful for student achievement when using feedback as part of an iterative design process. For student satisfaction, the study made no claims. For student support, the study found NBLE had significant student and faculty support-services. Finally, for administration, the study found the institution desired and backed modeling, prototyping, and iterative design.

Finally, Koory (2003) looked at undergraduate students in traditional and Web-based learning environments. Specifically, the study analyzed course management systems (CMS) that incorporated self, one-to-one, one-tomany, and many-to-many interaction. The study looked at student grades when making comparisons. Regarding student achievement, the study found students in asynchronous ISL environments performed better than those in traditional ones. In addition, online learning was found to be more amenable to adult learning styles and students tended to interact more with the content and the instructor than in the traditional environment. Regarding student satisfaction, the study found that students spent more time in lecture modes in traditional environments and that affected student motivation negatively. Students online were more satisfied in that they could pace themselves and better interact with the instructor. Discussion interaction was the biggest measure of success and satisfaction. For support services, the study found the instructor was heavily involved in supporting students in the online environment. No claims were made regarding the administration.

In summarizing the 13 studies indicating positive conclusions, the most significant findings included:

• institutional involvement was important to the success of ISL;

• technical support, design assistance, and training were key to student and faculty success operating in ISL environments;

• positive outcomes were reported when ISL was implemented programmatically rather than on a course-by-course basis;

• positive outcomes were reported when ISL utilized a feedback and redesign process;

• ISL provided a greater opportunity for student interaction with peers and faculty;

• interaction in discussions was important for measuring the success of ISL; and

• students showed increased motivation to participate in ISL environments.

Conversely, 16 studies (Arvan et al., 1998; Broad et al., 2004; Davis & Graff, 2005; Frederickson et al., 2000; Hammonds, 2003; Hiltz, 1997; Hiltz et al., 2000; Lim et al., 2006; Ostiguy & Haffer, 2001; Parker & Germino, 2000; Shapley, 2000; Sinyor, 1998; Wegner et al., 1999; Wegner et al., 1997; Witta, 2005; Yuan et al., 2006) reported the absence of substantial differences in major outcome measures both in purely online learning settings and between ISL and traditional learning environments. Like studies with positive results, NSD studies had proliferated. Additionally, NSD studies varied greatly in their methodology and the environments studied. As the purpose of this study was to provide a qualitative methodology for understanding possible trends emerging from the studies rather than to analyze their relative value independently, the following findings were not necessarily generalizable.

Following is a summary of basic demographic information for the 16 studies mentioned above. Figure 3, based on information included in the Appendix, reported that all the studies were spread over 10 years. Some 63% of the studies were published before the year 2001.

Studies also differed by the level of education and delivery models studied. Most (Arvan et al., 1998; Broad, et al., 2004; Davis & Graff, 2005; Frederickson et al., 2000; Hammonds, 2003; Hiltz, 1997; Hiltz et al., 2000; Lim et al., 2006; Ostiguy & Haffer, 2001; Parker & Germino; 2000; Shapley, 2000; Sinyor, 1998; Wegner et al., 1997; Yuan et al., 2006) studied the effectiveness of ISL environments in undergraduate education, whereas only two studies (Wegner et al., 1999; Witta, 2005) were specific to graduate education. Additionally, over half the studies (Arvan et al., 1998; Broad et al., 2004; Frederickson et al., 2000; Hiltz., 1997; Hiltz et al., 2000; Parker & Germino; 2000; Shapley, 2000; Wegner et al., 1997; Wegner et al. 1999; Witta, 2005) were actual comparisons of traditional face-to-face versus ISL delivery models and two studies (Hammonds, 2003; Lim et al., 2006) were comparisons of purely online and blended settings. Two other studies (Davis & Graff, 2005; Yuan et al., 2006) considered purely online learning settings only.

Studies used a variety of technological tools and media for course work. As the matrix shows, all the NSD studies used the Web and the Internet for course management purposes. Common themes that emerged from the analysis process included: (a) there were Web versions of courses; (b) the Web was used to mount course content, assignments, projects, and other information; (c) interactive multimedia was used in most but not all studies; and (d) text-based, asynchronous media was the predominant delivery-mode.

It was important to consider what sort of learning outcomes were addressed in these studies. Interestingly, as with positive conclusions studies, the effectiveness of ISL was evaluated by assessing a range of outcome measures. Figure 4 indicates the major outcomes considered. Student achievement as measured in grades and/or scores was the principal outcome measure in all the studies although some studies assessed additional measures. In general:

• all studies considered grades and/or exam scores not always converted to grades,

• some (19%) assessed student motivation,

• a quarter measured student satisfaction,

• over a quarter (31%) assessed student perception/attitude/feeling, and

• another 31 percent assessed course-related participation/interaction.

Although ISL tended to heavily rely on student interaction, support services were also reported. In most studies, student support existed in terms of: general training on technology use; technical help related to course Web sites; feedback provisions for general questions; and planning, monitoring, and evaluation of course-related projects. Tutors and/or instructors provided most support services.

Overall, the studies reported NSD results when the effectiveness of ISL environments was compared. Although some incremental changes were observed, particularly for student achievement, they were not significant. Thus, studies generally reported the absence of substantial differences in performance between ISL and classroom learning, purely online learning and blended learning, and before-after performances in online learning settings. The studies that additionally assessed motivation and satisfaction (see Appendix) found some level of motivation and satisfaction, although not significantly. Another pattern of themes emerged when studies were analyzed in terms of the nature and extent of student participation/interaction, including: (a) students took an active role in their learning, (b) instructors/tutors in some studies remained mere facilitators/organizers, (c) ISL tended to be more group-based than in face-to-face (f2f) learning, (d) ISL tended to be more feedback oriented than its f2f counterpart, (e) students used multimedia for interaction/participation/collaboration, and (f) ISL used both synchronous and asynchronous media though the latter dominated the former.

While findings were based on grouping all NSD results studies holistically by theme for analysis, outlining specific, exemplary research studies reinforced the strength of these findings. Four representative studies (Broad et al., 2004; Lim et al., 2006; Witta, 2005; Yuan et al., 2006) were selected to qualify these findings. These studies, selected because they were relatively recent, were summarized as follows.

Broad et al. (2004) studied the effectiveness of blended learning environments, of which there were very few such studies. The study considered the effectiveness of Web-based learning and teaching of an accounting degree course. Learning style, student perception, and the performance of 60 students were assessed. Pragmatist learning styles declined but reflective styles remained strongest. As reported, NSD was found in grades between previous cohorts of students and the same students taking another unit where there was no technology use in the latter two cases. The majority of students perceived the Web as a valuable technology for learning. With regard to support, a CD copy of the whole interactive learning environment was provided for each student at a nominal price. All lectures, exercises, model solutions, articles, and an electronic textbook were made available together with Web links. There were also links to a computer-aided learning package that could be accessed only within the university intranet.

Lim et al. (2006) focused on comparing the effectiveness of online and blended learning environments that enrolled respectively 59 and 69 Program Evaluation students. The study compared perceived versus actual learning and motivation and satisfaction outcomes among the groups. The online group was provided with Web-based content and the blended one attended to classes half online and half in the classroom. There was NSD in performance and motivation between the groups. However, significant before-after perceived and actual learning was found for both groups. Conversely, online students perceived: significantly less learning support, significantly difficult instruction, and significantly higher workloads than blended students.

Witta (2005) studied master’s-level educational research classes that used ISL environments. The purpose of the study was to compare performances of a traditional 15-week class, two eight-week intensive summer classes, and eight-week online summer classes on mid and final examinations. Ten homework assignments and a research proposal were given to all groups, although the nine assignments were group-based for the online class. PowerPoint presentations were used to highlight major points. All the groups sat for an identical 50-item multiple-choice mid-term exam. Additionally, all the classes sat for a 100-point open-book final exam, which covered similar content although not identical. Results showed that all groups performed equally in both exams. The online class scored highest in the final exam, although not significantly. The study made no report of other outcome measures and support systems or services.

Yuan et al. (2006) longitudinally studied social capital (interaction) and performance of 32 randomly assigned engineering students in a distance course for two consecutive semesters. Students were provided with Web-based course material and were provided with high-end laptop computers with access to the campus wireless network for the entire academic year. Two professors cotaught the course from two different universities via a coded audio/video system. E-mails, instant messaging, group discussion, document sharing, and other functions were supported by a special distributed-learning support system. NSD was found in achievement and social capital over time.

In summarizing the 16 studies indicating NSD conclusions, the most significant findings included:

• ISL research focused on undergraduate education;

• most research studies were comparisons of ISL and classroom learning environments;

• the Web was used to support course materials;

• communication/interaction was mainly supported by asynchronous tools;

• multimedia were used for course work;

• all studies used grades/exam scores as a measure of effectiveness;

• some studies additionally assessed student motivation, satisfaction, perception, and interaction;

• ISL was clearly participatory and student-focused;

• instructors assumed the role of a facilitator;

• ISL tended to be generally group and project based;

• ISL was more feedback intensive than classroom learning;

• studies examined the effectiveness of ISL across courses in the social and natural sciences;

• student support services were mainly provided by instructors/tutors; and

• NSD was found when ISL and classroom learning were compared in terms of grade achievement, motivation, and satisfaction.

There were many similarities between positive and NSD studies. Common major findings included:

• undergraduate education was usually the focus,

• technical support existed for students from beginning to end,

• the Web supported course materials,

• learning was student-centered,

• project and group-based learning approaches were chosen,

• multimedia supported interaction between students, content, and instructors,

• student engagement was feedback intensive,

• grade achievements were considered as the most important measures for success,

• comparison was mainly made between ISL and classroom learning environments,

• quasi-experiments and surveys dominated research methodologies,

• many studies appeared to lack clear theoretic underpinnings,

• most studies did not share consistent implementation or evaluation approaches,

• student support was mainly provided by the instructor, and

• all the studies were conducted in higher education institutions of the northern hemisphere.

Conversely, there were no clear patterns of differences between positive and NSD studies. However, the following findings could be highlighted:

• studies with positive conclusions indicated significant institutional support and a programmatic approach versus most NSD studies,

• NSD studies did not share consistent implementation or evaluation approaches,

• NSD studies lacked key theoretical frameworks and positive studies indicated some measure of a theoretical approach, and

• most NSD studies were conducted earlier than many positive studies.

Returning to the research questions, the review answered the following questions:

• How did the Internet affect learning in the environments studied?

• What methodological and theoretical issues characterized Internet-Supported Learning Research (ISLR)?

• What, if any, patterns and/or trends existed in these research studies?

• What implications existed for future research studies?

Using the criteria mentioned in the methodology section, 29 empirical studies were identified and reviewed. Slightly less than half (45%) reported ISL environments (ISLE) positively influence learning, whereas the rest (55%) reported NSD results. These results and the methodological and theoretical issues in research were discussed consecutively. The patterns and/or trends in research were highlighted in subsections. A more thorough discussion of these findings follows.

As mentioned before, the studies considered students’ grade/score achievements as the prime measures for success in ISLE. Thus, 45% of the studies indicated that ISLE positively affected achievement in higher education. Such learning environments also reportedly supported students’ motivation, satisfaction, and participation/ interaction/ communication. In short, ISLE was successful or effective in meeting student expectations. This finding was particularly important as most prior media-comparison studies documented only NSD results (Russell, 1999). Such a finding might partly reinforce the sustainability and scalability of ISLE. Moreover, these findings could partly justify the substantial investment in technology and related infrastructure. These findings seemed to be in congruence with the findings of a previous meta-analysis on the effectiveness of computer-based instruction (Waxman, Lin, & Michko, 2003). Generally, student outcomes were improved as a result of the integration of ISLE.

Contrasted to that, 16 of the 29 studies (55%) reported NSD findings as far as grade/ score achievement, motivation, and satisfaction outcomes were concerned. The interpretation of NSD results seemed to invite two opposing viewpoints among educational technology researchers. Some considered NSD results as “proof” for the effectiveness of technology-supported learning environments. This seemed to emanate from a preoccupation that such learning was less effective than classroom learning in meeting learning outcomes (Allen & Seaman, 2004; Russell, 1999). Others (Clark, 1985; Clark & Sugrue, 1988; Joy & Garcia, 2000) warned that these results should not be taken “at face value.” These researchers argued that most of the studies could have been flawed methodologically and, hence, other factors might have influenced the results. Generally, this argument seemed to reflect a more pessimistic attitude toward the ability of technology to better effect learning than traditional approaches.

Both the optimistic and pessimistic arguments seemed to take extreme sides. A more careful interpretation of NSD results was needed. According to Russell (1999), NSD findings indicated that technology-supported learning was neither more nor less effective than traditional learning. Thus, it made no difference whether courses were delivered online or on-campus as far as student outcomes were concerned. Instead, the choice between ISLE and classroom learning settings might have been affected by other factors such as convenience, affordability, infrastructure, and market pressures to use technologies (Clark, 1999; Russell, 1999). Both Clark and Russell further argued that technologies or media did not have any learning effect other than economic benefits. NSD results meant that although ISLE did not bring any measurable or substantial learning gains, these environments were at least as effective as classroom learning settings.

Historically, NSD findings were reported to represent the vast majority of educational technology or media-comparison research (Hill, Wiley, Nelson, & Han, 2004; Russell, 1999; Joy & Garcia, 2000). In the current review, however, nearly a competing number of SD and NSD results were found. Several issues could explain this difference in the quantity of NSD findings in earlier studies on technology-supported learning environments versus the current review.

First was the issue of technologic advances and sophistication. As indicated in the findings, ISLE was supported by modern technologies with unique capabilities compared to earlier technologies such as stand-alone computers, televisions, and radios. In the studies reviewed, the Internet provided the major technologic platform. Emphasis was given to anytime, anywhere learning relying on connectivity. The Web in particular was used as a source of rich and updated information. The review indicated that course content and expert opinions were all posted for students to access anytime. In this sense, it could be said that students learned from the Internet/Web technology (Hill et al., 2004). In other words, the technology was used as a vehicle for information. Additionally, the Web provided opportunities for interactivity and communication among students, course content, and instructors. In short, students learned through and with the technology (Hill et al., 2004). This aspect of technology use received much focus when studies were reviewed. Students were also active in communication and content creation.

Although both synchronous and asynchronous media were used for course-related purposes, the latter dominated contemporary ISLE both in SD and NSD studies. It was argued that such media allowed students to take more time for reflection and engagement. Students could access and participate in learning anytime and anywhere convenient to them. Again in most SD and NSD studies, course materials and activities were supported with multimedia such as text, audio, and video. Common to all the studies was that asynchronous, multimedia technologies allowed interactions between students and instructors. Hence, students were mostly learning from technologies in media-comparison studies whereas they were learning from, with, and through technology in ISLE. Consequently, it could be argued that the Internet and particularly the Web characterized the views of contemporary educational technology (Garrison & Anderson, 2003).

That asynchronous, multimedia, and interactive technologies influenced learning approaches was another pattern indicated in the review of ISLE. Unlike earlier technology-supported learning environments, which supported mainly instructor-dominated delivery, ISLE significantly required and supported the active role of students throughout course offerings. Common to both SD and NSD studies, there was a physical distance between students and instructors. In purely online settings, students did not meet physically, yet studies still reported the presence of online learning communities that assumed an important role in learning. The instructor’s role was limited to facilitating and organizing learning. As a result, student-centered learning was an important characteristic of ISLE.

A related pattern was that learning became project- and group-based more than ever. In most of the studies reviewed, students were required to work on course-related projects in groups. This approach brought learning to a different level. It was considered a way of sustaining student motivation and increasing achievement by creating opportunities for collaborative problem solving. A project-based approach would also assist students in developing the skills of planning, decision-making, monitoring, and evaluation: skills critical to professional development and contribution. In some cases, there might have been some “groupthink,” in which members were not able to adequately consider alternative approaches and solutions to problems other than their own.

Even though learning was student-centered, project-, and problem-based, there appeared to also be reasonable support from instructors and institutions. As the review showed, students were supported in various ways. There was reported support from departments and/or institutions in the form of increasing and improving the technologic ability to perform course-related activities. Students were also provided with multimedia and interactive technologies, although that differed across institutions and even across courses. What is important was that these institutional-level supports were reported in most positive results studies versus most NSD studies. In the former, a top-down and programmatic approach was available and there was thus more formal planning, implementation, and evaluation in positive result studies than in NSD studies. This was found to be a major difference between the two kinds of studies and a trend in ISLE in higher education. In short, institutions had begun to consider technologies as one of the major strategies to meet their missions and goals. Moreover, many institutions seemed to develop strategies to effectively and efficiently use technologies. As a result, learning technologies appeared to gain significant attention in instructional design, development, application, and evaluation. Compared to NSD studies, positive result studies reported a more pronounced institutional support and involvement in ISLE.

Another element of student support services common to both positive result and NSD studies was related to feedback. Because of the project and group-based nature of learning, feedback was required and provided in nearly all the studies reviewed. Obviously, the timing, extent, and relevance of feedback differed across courses and institutions. Feedback was used as a motivating and guiding strategy both at individual and group levels. Instructor support was also provided in the management of learning projects. All studies reported that instructors and/or tutors meaningfully participated in planning, monitoring, and evaluating student projects, indicating that instructors were still very much involved and needed in ISLE. Support also included initial user training on how to deal with ISLE and related technical issues, which were mainly provided by IT support personnel, tutors, and/or instructors. Generally, ISLE were well planned, executed, monitored, supported, and evaluated versus learning supported by older technologies. This could have contributed to student achievement, motivation, and satisfaction.

There appeared to be another difference between positive result and NSD studies of ISLE. Chronologically, more than half of the studies (63%) were conducted between 1997 and 2001 and dominantly had NSD findings. Starting in 2002, positive result studies appeared then to be more common in the research. In short, a trend in ISLE research was recent studies tended to indicate positive versus NSD results. This could be explained in several ways. First, ISLE, as mentioned in the findings, was more systematically designed, implemented, and supported than relatively older Internet and computer-supported environments. Institutional-level support could have been another significant factor. Another could have been that student and instructor skills, interest, motivation, and experience, which play a critical role in ISLE, were relatively better. Additionally, subsequent ISLE was more problem- and student-based. The unique capabilities of modern technologies discussed above could also partly explain this scenario. In sum, student support systems, technologic sophistication, and learning approaches characteristic of relatively recent ISLE could have led to better student outcomes.

As a pattern, most positive result and NSD studies (72%) considered courses at the undergraduate level. Graduate level ISLE was relatively limited. Perhaps most studies aimed at generalizing results to similar situations. This may have necessitated the inclusion of statistically large samples, which could easily be met at the undergraduate versus graduate level. A larger student population existed in undergraduate education than in graduate education for large-scale surveys. In addition, undergraduate programs had a larger pool of courses from which to choose, redesign, and offer with technologies. There could have been limited course menus in graduate-level studies. Moreover, there could have been practical issues in considering undergraduate courses for “experimentation.” Examinations at this level were focused more on measuring achievement as reflected by grades or scores. At the graduate-level, less emphasis was given to content mastery versus higher meta-cognitive processes that could generally be more demanding to support and assess. It could therefore be more cumbersome to evaluate ISLE in terms of meta-cognition versus cognition. Thus, convenience and affordance could have been some of the factors behind the significant use of undergraduate courses for experimentation.

Generally, nearly half of the studies indicated that ISLE was at least as effective as classroom learning environments as far as student outcomes were considered. The other studies reported that ISLE was more effective than traditional learning environments. These results of research should, however, be considered only cautiously and tentatively.

First, there was an implicit assumption in most studies that ISLE did influence learning. In actual terms, student achievement, motivation, and satisfaction were affected. Although there appeared to be some disagreement among psychologists concerning the definition of learning, most seemed to agree that learning was a process of “acquiring new and relatively enduring information, behaviour patterns, or abilities; modification of behaviour as a result of practice, study, or experience” (Corsini, 2002, p. 541). It was thus difficult to judge whether information gained or behavior modified as a result of ISLE would endure. Moreover, learning was broader and more complex than achievement, motivation, and satisfaction. As in the general area of educational evaluation, achievement was found to be the primary measure of success in ISLE. It was thus safe to conclude that ISLE influenced achievement and not learning as such.

Second, ISLE reportedly brought only limited levels and types of gains. As mentioned in the findings section, all the reported results were not conclusive and significant. There were NSD results as well. Even positive results were not conclusive. Third, considering the technological hype and the huge amount of investment in ISLE, a cultivation of only limited outcomes might not be a satisfactory achievement. Fourth, there were some qualitatively-driven studies, in which results could not be statistically and significantly tested. Some studies reported perceived learning, motivation, and satisfaction only. Fifth, even though much effort was made to include as many studies as possible for the review, only 29 were included. Some unpublished research reports that documented different findings might exist. Or, studies published in very local media outlets with different findings might exist. Finally, there were some more methodological and theoretical issues that needed to be considered in the analysis.

Some methodological and theoretical issues appeared to characterize research on ISLE. One pattern in research on ISLE was related to research designs most commonly used. Most (both positive results and NSD) studies employed quantitative designs such as experiments, quasi-experiments, and surveys versus qualitative designs. Several conditions might have encouraged or required such a dominant use of quantitative design in educational technology research. First and as mentioned previously, all the studies primarily considered grades or exam scores as measures of effectiveness. These measures were made available in quantitative terms. Second, there was at least an implicit assumption in all the studies that the relationship between ISLE and learning was causal, which could only be examined using quantitative designs such as experiments and quasi-experiments. Nearly half the studies reviewed were surveys. It was thus difficult to establish causal relationships between ISLE and learning. Even experiments and quasi-experiments did not sufficiently control “other factors” that could have confounded results. Third, the final aim of almost all positive results and NSD studies was to make generalizations, which were again assumed congenial only to quantitative designs. Fourth, such designs appeared to be more practical in the sense that they appeared to be less demanding in data collection and analysis stages than qualitative designs. Fifth, quantitative designs appeared to be common in the general area of educational research. Research in ISLE seemed to reflect this rhetoric. Generally, the aims of the studies and practical issues seemed to dictate the selection of quantitative approaches.

Another pattern in research on ISLE was related to the learning settings compared. When new technologies or methods were integrated in education, their effectiveness was usually measured in relation to the effectiveness of previous technologies or methods. Consequently, most positive results and NSD studies compared student performances in purely online and purely on-campus learning modes. The effectiveness of ISLE was compared to the effectiveness of classroom learning environments. Studies that used this approach received much criticism. Critics (Clark, 1985; Clark & Sugrue, 1988) argued that such studies looked like classic “apple-orange” comparisons. This argument seemed even more valid when reviewing less rigorous designs and more strict comparisons. Clark and Sugrue (1988) argued that teaching methods and other variables differ in the two learning modes.

According to the review, ISLEs were much more “pampered” than classroom learning environments. Literature indicated that cognitive performance, as reflected in student grades or exam scores, was enhanced by: collaborative learning (Gokhale, 1995); learner-centered, authentic, problem-based learning (Bradshaw, Bishop, Gens, Miller, &. Rogers, 2002; Scott & Brush, 1998); intensive training (Facione, Facione, & Giancarlo, 2000; Reed, 1998); and teachers’ cognitive skills or abilities (Paul, Elder, & Bartell, 1997). Similarly, student-centered and problem-based learning approaches improved problem-solving skills (Hmelo-Silver, 2004; Liu & Bera, 2005; Scott & Brush, 1998). As a pattern, almost all the studies reviewed integrated these important elements and approaches, which were deliberately withheld from the traditional learning environment. Consequently, comparing students’ performances in ISLE and classroom learning conditions seemed methodologically and theoretically debatable. These learning approaches did not appear to address the unique capabilities of technologies. They could have also improved classroom-learning environments if carefully integrated. Thus, results of comparative studies should be reported and interpreted cautiously.

Moreover, the use of theory in research on ISLE appeared to be less rigorous. Several studies did not use or clearly and sufficiently report the underpinning theoretical framework. Some studies appeared to adhere to the constructivist theory of learning without adequate explication. Consequently, designing, implementing, and evaluating learning projects did not appear to be systematically and rigorously conducted. This was more observed particularly in studies with NSD results. More positive results studies reported theoretical underpinnings than NSD studies. This was a major difference between the two kinds of results found in the review. The use of theory in the design, development, implementation and evaluation stages might have been more productive than the absence of a theoretical underpinning. Another theoretical issue common to positive results and NSD studies was that many of them did not operationally and consistently define constructs like learning effect, success, method, ISLE, blended learning, and online learning. Thus, the research process lacked standardization and coherence. This might have partly explained why much research was found to be inconclusive and unsatisfactory (Bates & Poole, 2003; Garrison & Anderson, 2003; Hill et al., 2004). Overall, the concern with rigor in educational technology research indicated concern about the absence of theoretical sophistication.

Such concern also appeared applicable to research on ISLE. Concern was especially related to the relative roles of instructional methods, media, contents, and technologies in bringing change in learning. This issue had been contentiously debated for some time. Generally, there were two contradicting viewpoints. First was the dissectionist approach in defining and experimenting with media, methods, contents, contexts, and technologies. This approach deemed it necessary to consider the relative effects of such variables on learning. It was argued that instructional methods, contents, and contexts—and not media or technologies—might have led to positive results in learning (Clark, 1994; Clark, 1996; Clark & Mayer, 2003; Russell, 1999). Contrasted to that, the holistic approach saw research approaches differently. It deemed any attempt to separately measure the effects of such variables on learning possible but futile (Bates & Poole, 2003; Kozma, 1991; Morrison, 1994; Reiser, 1994). Consequently, the systematic confluence of media, technologies, and methods as contexts positively influenced learning.

Unfortunately, the current review indicated few studies measured the effect of one or more variables on learning while sufficiently controlling the rest. Studies simply reported that technologies brought changes in learning. One could not be sure whether the reportedly positive or NSD results were due solely to certain assumed or even manipulated variables such as the use of Internet-supported technology. Alternative and competing explanations were not sufficiently ruled out in many studies. This seemed to invite “more skeptical eyes” to the reported results. In sum, theory development, application, and explicit report was overlooked in much educational technology research.

In addition, the external validity of results appeared debatable. All positive results and NSD studies were conducted in higher education institutions of the northern hemisphere with a particular inclination toward western institutions. A single research study was not found outside the northern hemisphere such as in an African context. This might have partly been due to the relative recency of ISLE to the southern hemisphere such as African campuses (Adam, 2003). Hence, there exists scant if any research on such learning environments. Previous research indicated that technologies were found to be culturally sensitive and restrictive (Chen, Mashhadi, & Harkrider, 1999; Joo, 1999; Frank & Toland, 2002). There were economic, cultural, social, and educational differences between the western world and the southern world such as Africa. The results reported so far regarding the effectiveness of ISLE did not address contexts outside the western world, which has clear implications for further research.

Many of the studies compared the effectiveness of fully online to classroom learning environments. Most higher education institutions seemed to reach a stage where the Internet supplemented some part of learning. On the other hand, classroom learning, which was an aspect of blended settings, was the mainstay medium in most of these same institutions. Researching the effects of the Internet in such blended modes seemed warranted. A more robust and interesting comparison could be made between blended and traditional learning modes, which are similar in some aspects, than comparing purely online and purely classroom learning contexts.

Because most successful studies indicated institutional support and a programmatic approach and most NSD studies indicated a lack of consistent implementation and evaluation, ISLE should have significant top-down support in a programmatic implementation. Those researching the effect of ISLE need to consider this aspect. Additionally, many NSD studies lacked key theoretical frameworks while many positive results studies indicated some measure of a theoretical approach. Thus, ISLE should be designed and studied with solid theoretic underpinnings. Explicit reporting of the methodological and theoretical aspects of research will better inform instructional design, further research, and educational practice.

The review indicated that ISLE employed student-focused, group-based, and project and problem-based learning approaches. Today, vast amounts of information across fields of studies are available online. Identifying relevant information is thus a challenge by itself. This is even more applicable in undergraduate education. Critical information literacy skills are required of contemporary students. Thus, an important question further research might ask is: To what extent should learning be group and project-based to be successful? Research should also examine the power of such approaches to bring desired change in ISLE.

Most studies were conducted at the undergraduate level. Graduate-level education studies were usually more student-focused and research intensive. Student-focused ISLE required significant use of modern technologies. Continuing to study the meta-cognitive, cognitive, and affective dimensions of learning in such environments over time is recommended.

Effectively studying higher-order thinking and affective dispositions such as motivation and satisfaction at the higher education level requires the use of a range of methods and approaches over a longer period of time. Thus, stronger analyses could be conducted by employing quantitative designs together with ethnographic approaches.

Because the studies reviewed were conducted in the northern hemisphere and because technology use is sensitive to cultural fabrics, the external validity of these findings to the southern part of the world such as Africa is questionable. To augment the impact of these findings, it is recommended that research on ISLE be conducted in such novel settings. This would not only strengthen the results of this review but would also extend our understanding of the effectiveness of ISLE across many cultures.

Although many important results were reported by this review, the authors have emphasized the following five critical findings:

1. The overall research indicated group and project-based learning approaches should be preferred.

2. Asynchronous, multimedia-supported delivery was most predominantly used.

3. The Web supported course content and delivery.

4. Student achievement was the principal outcome measure evaluating the effectiveness of ISL.

5. Top-down support structures and programmatic-based implementations indicated more positive results than those supported solely by instructors and/or tutors.