The best way to find things out is not to ask questions at all. If you fire off a question, it is like firing off a gun—bang it goes, and everything takes flight and runs for shelter. But if you sit quite still and pretend not to be looking, all the little facts will come and peck around your feet, situations will venture forth from thickets, and intentions will creep out and sun themselves on a stone; and if you are very patient, you will see and understand a great deal more than a person with a gun does. (Huxley, 1982, p. 20)
Editor, Distance Learning, Professor, Instructional Technology and Distance Education, Fischler College of Education, Nova Southeastern University, 3301 College Avenue, Fort Lauderdale, FL 33314. Telephone: (954) 262-8563.
Editor, Distance Learning, Professor, Instructional Technology and Distance Education, Fischler College of Education, Nova Southeastern University, 3301 College Avenue, Fort Lauderdale, FL 33314. Telephone: (954) 262-8563.
This marvelous quote from Huxley’s, The Flame Trees of Thika, illustrates a metaphorical rationale for a major refocusing of procedures for evaluation of distance education systems.
Of course, most distance educators agree that evaluation is an important step in the process of offering high quality distance education experiences—after all, it is the final step in every instructional design model. Often however, evaluation is either omitted from the instructional design process, or the results are filed away without being used. One reason may be that evaluation models are not intuitive or as readily applied as they could be. Stufflebeam’s context, input, process, and product model and Kirkpatrick’s four-level model come to mind (Kirkpatrick & Kirkpatrick, 2006; Stufflebeam, 2017).
An alternative model was first proposed by Fortune and Keith (1992). This approach is called the AEIOU approach for program evaluation, especially the evaluation of distance education projects. The effectiveness of AEIOU has been demonstrated a number of times (Simonson, 2005; Simonson & Schlosser, 1995a, 1995b; Sorensen, 1996; Sorensen & Sweeney, 1995, 1996, 1997; Sweeney, 1995).
The AEIOU approach is an eclectic one that uses quantitative and qualitative methodologies. It has two primary purposes as an evaluation strategy. First, the model provides formative information about the implementation of a project. Second, it provides summative information about the value of the project and its activities. The AEIOU evaluation process has five components and provides a framework for identifying key questions necessary for effective evaluation.
Component 1—Accountability
Did the planners do what they said they were going to do? This is the first step in determining the effectiveness of the program, project, or course and is targeted at determining if the project’s objectives and activities were completed. Evaluation questions typically center on the completion of a specific activity and often are answered “yes” or “no.” Additionally, counts of numbers of people, things, and activities are usually collected.
Questions such as the following are often asked to determine project accountability:
Were the appropriate number of class sessions held?
How many students were enrolled?
How many copies of program materials were produced, and how many were distributed?
Methods used: Accountability information is often collected from project administrative records. Project leaders are often asked to provide documentation of the level of completion of each of the project’s goals, objectives, and activities. Sometimes evaluators interview project staff to collect accountability data.
Component 2—Effectiveness
How well done was the program, project, or course? This component of the evaluation process attempts to place some value on the program, course or project’s activities. Effectiveness questions focus on participant attitudes and knowledge. Obviously, grades, achievement tests, and attitude inventories are measures of effectiveness. Less obvious are other ways to determine quality. Often, raters review course materials and course presentations to determine their effectiveness, and student course evaluations are used to collect reactions from distance education participants.
Examples of questions to determine effectiveness include:
Were the in-service participants satisfied with their distance education instruction?
Did the students learn what they were supposed to learn?
Did the teachers feel prepared to teach distant learners?
Methods used: Standardized measures of achievement and attitude are traditionally used to determine program effectiveness. Surveys of students and faculty can be used to ask questions related to perceptions about the appropriateness of a project or program. Focus groups (Morgan, 2018) also provide valuable information. Participants are systematically asked to respond to questions about the program. Finally, journals are sometimes kept by project participants and then analyzed to determine the day-to-day effectiveness of an ongoing program.
Component 3—Impact
Did the program, course, or project make a difference? During this phase of the evaluation, questions focus on identifying the changes that resulted from the program’s activities and are tied to the stated outcomes of the project or course. In other words, if the project had not happened, what of importance would not have occurred? A key element of measurement of impact is the collection of longitudinal data. The impact of distance education courses is often determined by following learners’ progress in subsequent courses or in the workplace to determine if what was learned in the distance education course was useful.
Determinants of impact are difficult to identify. Often, evaluators use follow-up studies to determine the impressions made on project participants; and sometimes in distance education programs, learners are followed and questioned by evaluators in subsequent courses and activities.
Questions might include:
Did students register for additional distance education courses?
Has the use of the distance education system increased?
Have policies and procedures related to the use of the distance education system been developed or changed?
Methods used: Qualitative measures provide the most information to the evaluator interested in program impact. Standardized tests, record data, and surveys are sometimes used. Also, interviews, focus groups, and direct observations are used to identify a program’s impact.
Component 4—Organizational Context
What structures, policies, or events in the organization or environment helped or hindered the project in accomplishing its goals? This component of evaluation has traditionally not been important even though evaluators have often hinted in their reports about organizational policies that either hindered or helped a program. Recently however, distance educators have become very interested in organizational policy analysis in order to determine barriers to the successful implementation of distance education systems, especially when those systems are new activities of traditional educational organizations, such as large public universities.
The focus of this component of the evaluation is on identifying those contextual or environmental factors that contributed to, or detracted from, the project or course’s ability to conduct activities. Usually these factors are beyond the control of the project’s participants. Effective evaluation of organizational context requires the evaluator to be intimately involved with the project or course to have a good understanding of the environment in which the project or course operates.
Questions typically addressed in evaluating organizational context include:
What factors made it difficult to implement the project or to successfully complete the course?
What contributed most to the success or failure of the program, course, project, or the students in the course?
What should be done differently to improve things and make the course more effective?
Methods used: Organizational context evaluation uses interviews of key personnel such as faculty or students, focus groups made up of those impacted by a program, and document analysis that identifies policies and procedures that influence a program or course. Direct participation in program activities by the evaluator is also important. Sometimes evaluators enroll in distance education courses. More often, a student is asked to complete a journal while enrolled in a course. By participating, the evaluator is confronted directly with the organizational context in which a program exists, and can comment on this context firsthand.
Component 5—Unanticipated Consequences
What changes or consequences of importance happened as a result of the program, course, or project that were not expected? This component of the AEIOU approach is to identify unexpected changes of either a positive or negative nature that occurred as a direct or indirect result of the program, course, or project. Effective evaluators have long been interested in reporting anecdotal information about the project or program that they were evaluating. It is only recently that this category of information has become recognized as important, largely because of the positive influence on evaluation of qualitative procedures. Often, evaluators, especially internal evaluators who are actively involved in the project or course’s implementation, have many opportunities to observe successes and failures during the trial-and-error process of beginning a new program. Unanticipated consequences of developing new or modified programs, especially in the dynamic field of distance education, are a rich source of information about why some projects are successful and others are not. Central to the measurement of unanticipated outcomes is the collection of ex post facto data.
Examples of questions asked include:
Have relationships between collaborators or students changed in ways not expected?
Have related, complementary projects been developed?
Were unexpected linkages developed between groups or participants?
Was the distance education system used in unanticipated ways?
Did the distance education system have an impact on student learning other than expected?
Methods used: Interviews, focus groups, journals, and surveys that ask for narrative information can be used to identify interesting and potentially important consequences of implementing a new program. Often, evaluators must interact with project participants or course students on a regular basis to learn about the little successes and failures that less sensitive procedures overlook. Active and continuous involvement by evaluators permits them to learn about the project as it occurs.
The AEIOU model is a dynamic one that permits the evaluator to tailor the process of program evaluation to the specific situation being studied. Those who use this model find it practical, straightforward, and effective.
And finally, “if you are very patient, you will see and understand” (Huxley, 1982, p. 20).
