Reevaluating Course Completion in Distance Education: Avoiding the Comparison Between Apples and Oranges

With funding tied closely to student enrollment and accreditation dependent on course quality, completion rates have become a significant measure in higher education. University administrators also watch completion numbers for their traditional programs closely as they analyze student progression or throughput to better balance student, staff, and faculty needs, numbers, and resources. In recent years, many of these same administrators have expressed interest in and concern about completion numbers for distance education courses and programs for a number of reasons. Debate also continues on whether and why courses for distance education students lead to higher noncompletion rates. If they really do have lower completion rates, some attribute the difference to the lack of faculty-student interaction, while others say it is impossible to compare the two groups because distance education students are inherently different from traditional students (e.g., older with additional commitments) (Carr, 2000).

This article provides a brief review of recent research on completion in distance education and exposes problems with the ongoing comparison between distance and traditional education completion rates. Problems include limitations in the research design itself, differences in student demographics, and inconsistent methods of calculating and reporting completion. These weaknesses suggest that educators may be metaphorically comparing distance education oranges to traditional education apples in ways that are invalid and unfair, especially whenever they also discredit distance education as a result. Our review and analysis of the literature indicates that not only is there little or no data extant on distance education completion rates, but what is available is suspect, as there is so much inconsistency in how the completion rates are calculated from courses to programs at institutions, and then from institutions in a region to those throughout the country. Hopefully, one of the reasons there is so little data available is because of research design problems that arise whenever researchers study self-selected—and not randomly selected—populations.

Arguably the most important part of this article is the review of literature which identifies some best practices to help ensure that distance learners have every opportunity possible to successfully complete their courses. It is evident from these research findings that some distance education students should not be expected to complete courses because of their life circumstances. Additionally, if completion rates are used as a criterion for evaluating the effectiveness of courses and programs, they are best done by comparing apples to apples and oranges to oranges at the specific course and program level, especially in the absence of any generally accepted algorithm for calculating completion rates.

Studies on distance education completion, especially those targeting online learning, are relatively few, due partly to the medium's relative newness. An article in The Chronicle of Higher Education in 2000 reported that

Some researchers have found that distance education completion rates are low—40 to 50% at best (Moore & Kearsley, 1996). (Moore and Kearsley's figures were given before the widespread use of online distance education.) However, “not all institutions are struggling as students and professors go online for the first time,” and significant variation exists among institutions, “with some reporting course-completion rates of more than 80% and others finding that fewer than 50% of distance-education students finish their courses” (Carr, 2000). In another study by Brigham (2003), 66% of distance-learning institutions have an 80% or better completion rate for their distance education courses, and 87% of institutions have 70% or better completion. Roach (2002) observed that “individual schools and organizations are reporting that their online programs have as high or higher rates of retention as their traditional classroom offerings” (p. 23).

While these studies reveal wide variance in completion rates, additional research focusing on the specific dynamics that influence completion and retention appear to be more consistent and helpful. Kemp (2002) found that “the adult distance learner may be affected by a variety of internal and external factors that account for the continuance/discontinuance in their studies” (p. 65). He cites studies by Kennedy and Powell (1976) and Brindley (1987) which state that “life circumstances combine with other factors (e.g., independence, organizational abilities, and social support) as predictors of persistence or withdrawal” (p. 65). Kemp also refers to a study by Powell, Conway, and Ross (1990) that reported: “Life circumstances interact with predisposing characteristics (e.g., educational preparation, socioeconomic and demographic status, and motivational and perseverance attributes) to influence completion.”

Some other interesting factors have been identified to help predict distance education completion rates. A study by King found that students who had taken a distance education course were “more likely to recommend distance education,” and that “successful completion of other distance education courses is a good predictor of students who are likely to complete subsequent courses” (King, 2001, p. 414). Kemp (2002) pointed out that a number of studies show that previous experience with distance education is associated with greater retention and lower frequencies of dropout (Coldeway, 1982; Langenbach & Korhonen, 1988; Rekkedal, 1983, as cited in Kemp, 2002). As Wlodkowski (2003) asserts, “Prior college experience may provide some degree of confidence, coping skill, and familiarity with college learning, contributing to successful persistence and degree attainment” (p. 11). Wlodkowski's study found that “better social integration with peers correlates with persistence,” and he notes that “research findings from other studies confirm that positive involvement with peers and faculty encourages adult students to persist” (New England Adult Research Network, 1999; Tinto, 1998, as cited in Wlodkowski, 2003, p. 12).

Comparisons between traditional and distance education are difficult for many reasons, including inconsistent methods of calculating and reporting completion, differences in student demographics, and the limitations of research studies. Despite the research findings on completion that do exist, many still express the reasons for completion rates being what they are as “complex and poorly understood” (Jackson, 2001, p. 3). Henke and Russum (2000) argue that “there is a lack of validated variables or frameworks to measure attrition within distance education courses” (Sheets, 1992; Thompson, 1997; and Parker, 1999, as cited in Henke & Russum, 2000, para. 11). Due to the absence of a standardized measurement of retention, researchers should use caution in making unqualified statements about whether poorer completion rates are caused by lack of quality in distance education, because many outside variables can confound such results. For example, one study found a high percentage of noncompleters in a class even though the “students who completed … liked the online approach, learned the material, and spoke highly of the experience, despite the attrition rates” (Zolkos, 1999, as cited in Henke & Russum, 2000, para. 10).

The widespread belief that completion is lower in distance education does not seem well-founded; there is no national standard for calculating completion rates for both traditional campus-based and distance education programs. Many distance education administrators warn against comparing “the statistics of different institutions, since they measure completion rates differently. Some institutions, for instance, don't include in their dropout calculations those students who leave classes during drop/add periods at the beginning of a semester, while others do” (Carr, 2000, para. 11). One example is the study by Kemp (2002) in which noncompleters were defined “as those students who (a) were nonstarters—that is, they did not commence work on their course; (b) withdrew from their course; or (c) received an academic failing grade” (Bajtelsmit, 1988, as cited in Kemp, 2000, p. 68). Using this algorithm for calculation, many of the noncompleters could simply be nonstarters, who did not lack motivation to finish the course because they never started the course. A good example of this clarification is Kemp's study itself: of the 64 students who did not complete the course, 38 did not commence work on the course, 19 withdrew from the course, and 7 received an academic failing grade.

At the author's own university and independent study program, nonstarters in traditional courses are not considered dropouts, although they are considered dropouts for distance education courses. This kind of inconsistency and confusion in reporting methods has university administrators everywhere comparing apples to oranges, leading them to sometimes prematurely and unfairly disparage the quality of distance education. From an internal report (September 2000 to August 2001) at this same university's distance education program come the following three examples of what completion rates would be if nonstarters—students who either change their enrollment or drop the course within the first 2 weeks of enrollment—were not included:

• The first course shows an initial completion rate of 36%. However, 56 of the 93 students who enrolled never submitted a lesson. When these 56 students are excluded, the completion rate is 97 percent.

• The next course shows an initial completion rate of 76%. However, 72 of the 427 students who enrolled never submitted a lesson. When these 72 students are dropped, the completion rate is 92%.

• Another course shows an initial completion rate of 71%. However, 108 of the 518 students who enrolled never submitted a lesson. When these 108 students are eliminated, the completion rate is 91%.

Besides consistency in calculating course completion, there must also be uniformity in calculating program and degree completion. Currently, the calculation of degree/program, not course, completion rates for on-campus students is generally found through graduation rates reported to the U.S. Department of Education as part of the Integrated Postsecondary Education Data System reports. This is a 6-year graduation rate for a cohort of first-time, first-term entering freshmen. For example, this year many universities will report a graduation rate for the 1997 Fall class of entering freshmen as of August 2003. Retention rates are calculated based on those students who return to campus for their sophomore year. Can one imagine distance education programs being held to the same 6-year period used for on-campus students when so many distance education students are part-time?

In addition to calculation and reporting inconsistencies, differences in student demographics make completion comparisons troublesome. Distance education students are generally older, with more work and family responsibilities. Receiving an education for these students is clearly a luxury that must be put on hold when other real-world commitments or difficulties arise. As institutions make efforts to improve completion, administrators and faculty should remember that adult students are “most likely to leave [the] university because of ‘facts of life’ reasons, the most significant and influential of which is finance” (Bolam & Dodgson, 2003, p. 181). A study by Wlodkowski (2003) similarly found that lack of time was the primary reason for student attrition among adults. The adult students in this study “repeatedly and emphatically mentioned competing priorities and not having enough time to meet the demands of family, work, and school. [For these adults] the top two reasons for leaving college indicated in the survey were ‘conflict between job and studies’ (60%) and ‘home responsibilities too great’ (59%)” (p. 11).

A number of distance education administrators concur with these findings. Jacquelyn B. Tulloch, while executive dean of distance education and college services at the Dallas Community College's LeCroy Center for Educational Telecommunications, commented, “Distance education students tend to leave us because they are very busy, their lives are crammed full of things, and suddenly they find themselves in a situation of having to rethink their priorities” (Carr, 2000, para. 11). Another administrator reported that two thirds of students who leave an independent general studies program do so for personal reasons or time constraints, and most of these students say they will return when these obstacles are sufficiently resolved (Allred, 2003).

Many administrators see attrition as failure, but Diaz (2002) argues otherwise:

Because adult learners generally have more responsibilities that make it harder to complete their courses, the criticism of “lower” completion rates for distance education—if they are really lower—should be mitigated. Jackson (2001) asserts that distance education actually improves overall completion by helping retain students who otherwise would have discontinued their studies. He cites the U.S. Department of Education's (1999) findings about seven situational factors which play a significant role in determining whether students will persist in college:

1. Delayed enrollment in college

2. Being a recipient of a GED

3. Being financially independent

4. Having children

5. Being a single parent

6. Going to college part-time

7. Working full-time during college (p. 3)

He asserts that distance education facilitates access and is convenient for many with these characteristics, namely those who have children, are going to college part-time, or are working full-time during college. Researchers and analysts should evaluate whether the benefits of offering access to some education at a distance outweigh the limitations of less faculty interaction in many of these courses and access to “no” education at all. Distance education must certainly help more students to complete who would never complete otherwise.

Even if distance education generally benefits nontraditional students by providing access and convenience, traditional students still enroll in these distance courses and may even confound some of the completion data. Jackson (2001) contends that “there is little doubt that Web-based or even interactive video courses present a daunting challenge to undergraduate students whose discipline and motivation may not be sufficient to complete such courses” (p. 4). At times these less motivated students from the traditional setting may sign up for distance courses so they do not have to go to class. Unfortunately, when they do not finish the course, the medium is often blamed for a consequence actually caused by the students’ characteristics. Visser (2002) found that in distance education “it is often motivational problems, and not the instruction itself, which lay at the root” of low completion rates (p. 95).

Inconsistencies and limitations in research design and studies are two other critical reasons that make comparisons difficult. Furthermore, “distance education research has unjustly faced a higher burden of proof than other scientific and educational research” (Brown & Wack, 1999, as cited in Diaz, 2000, p. 26). The problem of self-selection also complicates comparison studies, in that the studies rarely (if ever) divide subjects randomly between traditional and distance education. This makes it is impossible to know whether differences are caused by student characteristics or delivery formats. Jackson (2001) notes an additional area of limitation, stating “although most studies show that a high degree of interaction is absolutely necessary for a successful distance education course, there is seldom an empirical comparison of this requirement for distance education course with their in-class equivalents” (p. 4).

One point of possible confusion is the distinction between the completion rates of individual distance courses and the completion rates of programs utilizing distance courses (Dallas, personal communication, 2003). For many programs, “it is still too early to compile statistics on the retention of students in degree programs offered through distance technologies. Instead, they focus on individual course-completion rates” (Carr, 2000, para. 12). By not focusing on degree completion rates, researchers cannot determine whether many online students are simply switching to more convenient classes so they can finish their degrees. “Managing the tension between shortterm course persistence and longer-term degree persistence is a challenge that all universities face,” and distance education administrators should be proud their programs “have the potential for addressing some … of the reasons why some students do not return to colleges or universities” (Jackson, 2001, p. 5).

A final area of difficulty for comparing distance education with traditional formats was noted by Saba (1998), who observed that researchers often do not clarify what they mean by traditional education. Does this mean a lecture format, one that is discussion based and student-centered, or something else? Completion and retention rates can vary as widely among “traditional formats” as they can between distance formats (e.g., correspondence, video, Internet). By making simple comparisons between traditional and distance education, researchers incorrectly imply that all types of instruction in these two formats are exactly the same.

The inconsistencies that arise when completion rates between traditional and nontraditional education are compared reaffirm the necessity of constructing a standard of measuring retention, at least among the same types of delivery formats, for example, traditional and distant. Because of the relationship that enrollments have to funding, accreditation, and university reputation, accurate information on completion rates is a minimum expectation for educators.

The integration of standardized methods for calculating completion rates, along with the use of the following suggestions, would increase effective practices for the improvement of both traditional and nontraditional students’ completion and retention rates.

Many articles containing completion information have only a few lines on these topics, usually mentioning that rates have gone up or down at an institution and describing how this was accomplished. A number of scholars and researchers, however, have identified strategies to increase completion in distance education courses. We have limited our discussion here to the most helpful suggestions and practices identified from the literature.

Many articles cite the need for an orientation program before an online program begins. “Orientation for online courses serves the same objectives as orientation for college, in the sense that it can facilitate academic and social interactions, increase student involvement, enhance the sense of belonging to a virtual learning community, and help retention” (Scagnoli, 2001, p. 20). Universities such as San Diego State prepare distance education students by providing them with information about the “modes of delivery and technological requirements of each course, program, and degree offered by the university” (San Diego State University Strategic Plan, p. 2). Diaz (2002) suggests that distance education administrators encourage teachers to give more attention to students’ readiness before a distance class with skill surveys and through pinpointing reasons for student success, providing online orientation courses and help desks to reduce drops related to technical difficulties, and provide pedagogy-based training to broaden online teachers’ experience and familiarity with technical teaching tools (p. 4).

Some experts consider learning motivation to be “more important in distance education courses than in conventional courses, because distance learners with low motivation have more of a tendency to drop out or fail” (Jung, Choi, Lim, & Leem, 2002, p. 160). “Moore (2001) noted that to be successful in delivering online courses, faculty must … provide specialized attention to students with low levels of self-directedness” (Lindner, Dooley, & Kelsey, 2002, p. 2). “Establishing some form of personal contact with students and letting them know what is required in a distance course are both essential. Successful instructors frequently give their often overloaded students some flexibility in assignments and testtaking” (Carr, 2000, para. 9). Motivation and retention problems may be mitigated by mentoring and other encouraging social factors. Student-student and faculty-student interaction can be critical to the perseverance of struggling students. It is often the students who are having the most problems who “do not have the confidence to approach university staff” (Bolam & Dodgson, 2003, p. 187), so faculty and staff must take the initiative and reach out to these learners.

Because completion appears to be linked to faculty and staff interaction with students, institutions cannot increase their enrollments and expect to improve completion without adequate faculty and staff support for students. More faculty or fewer enrollments keep class sizes small, which is another completion and retention strategy suggested by many experts. In a 2000 study by the National Education Association (NEA), 66% of NEA faculty had an enrollment limit on their courses, and faculty with enrollment-limited courses felt more favorable about distance learning than those without limits, regardless of how low or high the limit. A study by Brigham (2003) found that 60% of instructor-led courses have 20 students per faculty member, and 32% have 30 or more students per faculty member. Small classes make it easier to build community, which is significant because “students who feel connected to other students and campus community are more likely to persist to graduation” (Astin, 1993, as cited in Scagnoli, 2001, p. 24).

Mary Hricko of Kent State University suggests seven steps to improve completion and retention, which administrators can compare to their current practices to determine areas for improvement:

1. provide students an orientation to the course;

2. allow students to engage in a formative assessment throughout the course;

3. educate students that the technology and content of the course are two different elements of the course;

4. bring campus life to the class;

5. be innovative and establish an online learning community;

6. be available; and

7. partner students with study-buddies (Hricko, 2003).

Kasworm (2002) similarly provides an extended list of strategies for reducing attrition and improving completion, especially among adult learners, which includes the following:

1. provide initial entry advisement, orientation, and career counseling;

2. offer financial assistance or financial counseling;

3. provide academic and basic skill development opportunities;

4. establish policies and procedures oriented to adult learners;

5. use information technology (e.g., listservs, online forums) to build community;

6. establish programs that incorporate family and spouse support;

7. increase opportunities for personal interaction with and attention from faculty;

8. provide assistance in finding specialneeds services, such as housing, transportation, and so forth;

9. establish adult support networks; and

10. get to know and treat students as individuals.

Some strategies to foster higher completion rates are facilitated by the Internet, one example being instructors’ ability to track students’ online activity, noting how often they log in to the course and how much time they are spending. This tactic is not possible in traditional classes, where it is hard to know how much time students are putting into coursework outside of class. To minimize student problems, Otton (2003) advises administrators to train instructors to develop a set of frequently asked questions online, which can also “reduce the additional time that might be necessary to help students on an individual basis” (p. 29). With such methods available, it is not surprising that a study by Roach (2002) found that dropout rates are higher when the instructor has not been trained in how to teach online.

Distance education administrators should at least set their own goals to improve completion and minimize attrition of their distance education students while engaging university counterparts and national associations and accreditation agencies in a discussion about establishing consistent and uniform methods in calculating and comparing completion rates for both campus and distance education students. Guidelines from the American Council on Education (2000) state that administrators should answer the following questions:

It should be restated that even the best practices for improving completion rates do not always work for each institution—and that is to be expected, especially when considering the diverse needs and characteristics of adult students. On the other hand, many instructors find that as they gain more experience teaching distance education courses, they are better able to influence completion rates in their courses (Carr, 2000).

Academe is in a time of flux and transformation as students, faculty, and administrators adjust to distance education's emerging role within higher education. The tendency to compare completion rates for courses and programs in both traditional and distance education contexts, while natural, is problematic. The unique characteristics, needs, and motivations of students who self-select the distance format from those who self-select the traditional format are not easily compared, as good research design depends on randomly selected and not self-selected subjects. Furthermore, no standardized algorithms for calculating completion rates currently exist in higher education generally and in distance education specifically.

Since “colleges are moving toward a point where students may be matched with a particular delivery medium based on their learning styles and on their lives” (Carr, 2000, para. 29), it makes more and more sense to compare apples with apples and oranges with oranges. This may mean comparing the characteristics of the last crop of apples with the current crop—and the same with oranges—all within the same institutional orchard or family rather than insisting on still trying to compare the traditional education apples with the distance education oranges. As important as the emergence of standards for calculating completion rates within a certain type and format of class could be, for example, completion rates for a previous and current distance education course, the more important effort may be in shifting the interest in comparisons to the emphasis on identifying and promoting best practices for improving completion rates. After all is said and done, all researchers and practitioners can never forget that each completion and noncompletion statistic represents a real student with unique needs, interests, and motivations. To judge a course and the quality of a program by a student's not completing the course is premature and inconclusive, especially whenever the reasons for attrition have nothing to do with the course itself. Whether the apples complete more than the oranges is not as important to these individual students as them knowing that every opportunity for their success and completion has been afforded them by their institution's adoption of best practices.