Evaluation procedures are becoming of critical interest to trainers and teachers who are adopting e-learning or distance education (Peak & Berge, 2006). As new distance education systems are being planned and implemented, there is considerable concern that the time and effort required to move to distance delivery of instruction produces a valuable educational experience; thus, evaluation is regularly a part of plans to move from traditional face-to-face instruction to distance education. Kirkpatrick’s (1998) evaluation approach with its four levels of evaluation, supplemented by Phillips’ (2002) fifth evaluation level—return on investment (ROI)—seems to be the preferred approach of many trainers, and some educators.
Kirkpatrick’s evaluation approach has been traditionally used to evaluate classroom training and teaching, especially in the private, government, and military sectors. It is a straightforward approach that produces usable information for the trainer. The four levels of the approach are designed to obtain answers to commonly asked questions about training: Did they like it? Did they learn it? Will they use it? Will it matter?
LEVEL 1: REACTIONS (DID THEY LIKE IT)
As the word “reactions” implies, evaluation at this level measures how participants in the training program feel about the educational activity. Students are asked what they liked and did not like about training, sometimes several times during a course or program. Students are required to use checklists, Likert responses to statements, and open-ended comments, all to determine if the training was perceived positively by participants.
LEVEL 2: LEARNING
At this level, evaluation strategies attempt to determine more than learner satisfaction. Rather, evaluators assess the extent to which learners have advanced in skills, knowledge, or attitude. What and how much did participants learn? What new skills do they possess? And, what new and appropriate attitudinal positions have been produced?
Methods include objective testing, team assessment, and self-assessment. Often pre-test-posttest change is used as a measure at level 2.
LEVEL 3: TRANSFER
At this level, evaluators attempt to determine if the skills, knowledge, and attitudes learned as a result of training are being transferred to the workplace or to actual learner activities. Evaluation questions deal with the use of new skills, or the application of new knowledge to events. Timing of the evaluation at this level is critical, and problematic, since it is difficult to know when transfer actually occurs.
LEVEL 4: RESULTS
Evaluation activities at this level attempt to measure the success of the training or teaching program in terms of increased productivity, improved quality, lower costs and, for businesses, even higher profits. Trainers are increasingly being asked to demonstrate the direct and indirect impact of training on the success of the organization and to relate training to mission accomplishment. In schools, level 4 evaluations often look at enrollments in additional courses, learning motivation, and educational achievement.
LEVEL 5: RETURN ON INVESTMENT
Increasingly, many training and educational organizations that are adopting e-learning and distance education are interested in the concept of return on investment—converting training results from e-learning activities into monetary values and comparing these costs to the cost of the training program to determine a return on investment. Phillips (2002) describes a five-step process to determine return on investment.
First, it is necessary to collect level-4 data to determine if there is a change in job or educational performance that is positive and also measurable. This assumes that there were evaluation data collected concerning the first four levels of the Kirkpatrick model.
Second, evaluators need to identify the training that contributed to the change in performance. Testing can be used, as can control groups that receive different training, or no training at all.
Third, it is necessary to convert the results of training or education into monetary values. This often means a relatively subjective process must be undertaken to quantify outcomes related to the training.
Next, the evaluation process requires the determination of the total cost of training. This includes trainer costs, facilities expenses, materials purchased, and other expenses.
Fifth, ROI is determined by comparing the monetary benefits to the costs. In this manner, it is possible to quantify the impact of training, the effectiveness of education, and the value of the instruction.
The ROI process is time consuming, requires a skilled evaluation team, and is sometimes criticized because it produces evaluation results that look at what has happened, rather than what will happen. Peak and Berge (2006) also note that not everything needs to be measured. Rather, leaders determine what they think is important and then trainers evaluate those areas.
While evaluation has always been somewhat important in corporate and military training and of interest to a lesser extent in education, the recent phenomenal growth of e-learning and distance education has made many leaders want to know what the implications are of moving to training and teaching that is not face-to-face. Thus, Kirkpatrick’s and Phillips’ evaluation approaches have received increased attention, especially since most evidence clearly demonstrates distance education works academically to produce required achievement gains. The evidence is clear that students learn just as effectively when they are taught at a distance as compared to when they learn in a traditional classroom (Simonson, Smaldino, Albright, & Zvacek, 2006). Thus, it can be generalized that traditional training and e-learning work equally well. The question for evaluators, then, becomes the determination of the advantages, if any, of moving to an e-learning environment. Evaluators are looking at cost savings, time savings, increased motivation and satisfaction, economies of scale, and other non-achievement outcome metrics. Evaluation of e-learning should provide leaders evidence they need to support or to refute training decisions.
