The purpose of this research was to manipulate the component of confidence found in Keller’s ARCS Model to enhance the confidence and performance of undergraduate students enrolled in an online course at a Texas University. This experiment used SAM Office 2003 and WebCT for the delivery of the tactics, strategies, confidence-enhancing e-mails (CEE), and course content. The students were trained to use the Microsoft Access database program for 5-and-one-half weeks. The results indicated that the treatment group showed no statistically significant gains over the control group for the variable of learner confidence. The treatment group did statistically significantly better than the control group on the posttest.
Introduction
Motivation is a highly important aspect of learning. Means, Jonassen, and Dwyer (1997) cited studies showing that motivation accounted for 16% to 38% of the variations in overall student achievement. However, an extensive review of the literature leads one to concur that there is a noted lack of research concerning the motivational needs of learners (Astleitner & Keller, 1995; Gabrielle, 2003; Means et al., 1997; Shellnut, Knowlton & Savage, 1999; J. Visser & Keller, 1990).
This is particularly true with computer-based and distance instruction. Keller (1999a) noted that self-directed learning environments posed greater challenges to learner motivation than their face-to-face counterparts. Song and Keller (2001) advised that continued problems with learner motivation in Web or site-based computer-assisted instruction (CAI) were often the result of incorrect assumptions on the part of instructional designers that motivation, if taken into account at all, was assumed to be already present in the CAI. They also noted that with the widespread use of computers in education, one could no longer depend on the “novelty effect” of technology to stimulate learner motivation.
Distance education environments provide unique challenges for instructors and designers who wish to motivate students. Traditional distance learning models stress the independence of the learner (Downs & Moller, 1999; Moore, 1989) and the privatization of the learning environment (Keegan, 1986; Moller et. al. 2005). Such student-centered, independent learning requires a strong sense of motivation and confidence.
Keller’s ARCS Model and Previous Studies
To stimulate and manage student motivation to learn, Keller (1987a, 1987b, 1987c) created the ARCS Model of Motivation. ARCS stands for Attention, Relevance, Confidence, and Satisfaction and serves as the framework for the confidence-enhancing tactics found in this study.
Keller’s ARCS model enjoys wide support in the literature, and a number of researchers attest to its reliability and validity in many different learning and design contexts. The ARCS model is an attempt to synthesize behavioral, cognitive, and affective learning theories and demonstrate that learner motivation can be influenced through external conditions such as instructional materials (Moller, 1993). The ARCS model was initially predicated on the expectancy value theory based on the work of Tolman (1932) and Lewin (1938). The expectancy value theory essentially states that learners pursue activities they value and in which they expect to succeed (Keller, 1987c).
ARCS research can be found concerning the traditional classroom (Bickford, 1989; Klein & Freitag, 1992; Means et al., 1997; Moller, 1993; Naime-Diefenbach, 1991; Small & Gluck, 1994; J. Visser & Keller, 1990), computer assisted instruction (Asteitner & Keller, 1995; Bohlin & Milheim, 1994; ChanLin, 1994; Lee & Boling, 1996; Shelnut, Knowlton & Savage, 1999; Song, 1998; Song & Keller, 1999; Suzuki & Keller, 1996), blended learning environments (Gabrielle, 2003), and online, distant, and Web-based classrooms (Chyung, 2001; Huett, 2006; Song, 2000; L. Visser, 1998). In fact, Means et al. (1997) called Keller’s ARCS model the “only coherent and comprehensive instructional design model accommodating motivation” (p. 5).
While Keller (1987a) believes that “motivational interventions can be focused within a general category or specific subcategory of the model” (p. 6), there is insufficient evidence to support claims that learner motivation can be isolated or compartmentalized into separate categories. Studies of ARCS-enhanced instructional materials have returned inconsistent results on the individual subsections as well as on the overall measure of learner motivation (Babe, 1995; Gabrielle, 2003; Moller, 1993; Naime-Diefenbach, 1991). In addition, Means et al. (1997) found that “there is inconsistent evidence that each of the factors operates independently, that learners’ motivations can be decomposed and isolated, or that changes in one motivational state have an inconsequential effect on others” (p.6).
Confidence
Confidence has been described as an inherent personality trait (McKinney, 1960). However, confidence is generally accepted as situation-specific, and it can therefore be manipulated by internal and external factors (Keller, 1979; Moller, 1993). In his development of social learning theory, Rotter (1954) argued that people have a tendency to ascribe their failures or successes to internal or external factors: he found that people tend to pursue that which brings about the most rewarding consequence, which he called expectancy.Bandura (1977, 1986) elaborated on this concept when he explained that individuals’ expectancy is related to their estimate of the outcome of a given behavior. He used the term self-efficacy to describe the belief that one’s abilities and knowledge are sufficient to be successful at a given task: learners who expect to succeed demonstrate more confidence than learners who expect to fail. However, a learner may still possess confidence without the guarantee of success, as long as the challenge is “within acceptable boundaries” (Naime-Diefenbach, 1991, p. 12).
Building on the work of Rotter, Bandura, and others, Keller defines confidence as “helping the learners believe/feel that they will succeed and control their success” (Keller, 1987a, p. 2). Confidence is the interplay between desire for success and fear of failure. These opposing forces vie for control over the learning experience. To better understand the role of confidence in the ARCS model, it helps to examine what Keller and Suzuki (1988) characterize as its three most important dimensions: perceived competence, perceived control, and expectancy for success.
Perceived Competence
The extent to which learners feel they are able or unable to learn is perceived confidence. Learners who believe in their potential success are more likely to exert the effort required to be successful. Despite the fact that learner expectations are not always realistically aligned with their abilities, expectations can still positively influence outcomes (Bickford, 1989).
Students with a poor perception of their abilities may become anxious and perform less well than their counterparts with higher confidence in their abilities (Naime-Diefenbach, 1991). Moller (1993) describes learners with high anxiety as often “misdirecting effort from learning to task-irrelevant concerns. Learners high in anxiety are often low in self-esteem and, as such, avoid evaluative situations” (p. 7). In contrast, learners with normal anxiety levels feel more confident and motivated in situations where they must be evaluated.
Perceived Control
When learners believe their efforts and decisions have real consequences, they feel more confident (Bandura, 1977; Keller & Suzuki, 1988). This fosters a higher internal locus of control and a greater sense of self-pride and accomplishment (Moller, 1993). In contrast, learners who believe luck or other uncontrollable outside forces are in charge of their successes or failures tend to feel more helpless and unconfident, and perform at lower levels.
According to Keller and Suzuki (1988), “features in the instruction that promote feelings of personal control over outcomes will help develop confidence and persistence” (p. 405). This is supported by researchers such as Carroll (1963), Bloom (1976), and Kinzie and Sullivan (1989), who recommend allowing learners to control the pace of instruction. However, research is mixed about how much control is actually beneficial to learners (Klein & Keller, 1990). Steinberg (1989) cited numerous studies showing that learners with little prior knowledge of the subject matter were likely to perform poorly with increased learner control.
Keller (1987a) suggests one strategy for fostering control is to give students knowledge of what is expected of them. However, this is not enough to guarantee confidence: while learners may understand what steps are necessary to complete an assigned task, without the confidence in their ability to successfully complete those tasks, they may not perform as well as they should (Moller, 1993).
Expectancy for Success
Learners’ expectations or beliefs can influence outcomes. If the learner believes he will be successful at a given task, such belief may result in greater effort expended and improve success. Learners with such expectancy for success also possess higher motivation than learners who expect failure (Naime-Diefenbach, 1991). Conversely, the learner who expects failure may evince learned helplessness (Keller, 1979; Seligman, 1975). Once taken hold, learned helplessness can be a powerful impediment to success.
In summary, in order to help learners overcome learned helplessness and other self-fulfilling prophecies, it is necessary for instructional designers to consider learner anxiety and provide for instruction that helps boost learner confidence, making them feel competent, in control, and successful. While fear of failure can strongly affect motivation in traditional learning environments, it may be an even greater factor in distance education (L. Visser, 1998). Even with highly motivated students, the isolation of the learner, an unfamiliar distance environment, the technology required in distance courses, and the distance separating learner and instructor have an effect on learner confidence. The concept of perceived control may be particularly relevant to distance learning environments. Roblyer (1999) found that students who chose distance education classes over face-to-face classes often did so out of a greater desire or need for control over their own learning outcomes. Studies have shown that technology brings with it new attitudes and anxiety levels that can have a direct effect on confidence (Yaghi & Ghaith, 2002). The instructor of the distant course may need to be especially concerned with increasing and maintaining learner confidence.
The major focus of this study was to determine whether confidence could be specifically targeted for improvement and whether improvements in confidence would translate into performance gains. The underlying assumptions are that confidence is a highly important aspect of motivation, that it can be manipulated through external factors, and that it has an effect on learner performance.
Method
This study was conducted over a period of approximately 5-and-one-half weeks. The purposes of this research were to: (a) determine if there were statistically significant increases in confidence levels of online learners using systematically designed confidence tactics based on Keller’s ARCS model; and (b) determine if the tactics also produced a statistically significant increase or change in academic performance.
Within the ARCS model, confidence can be increased (Keller 1987a, 1987b) by examining learning requirements (LR) to give students knowledge of what is expected of them. Confidence can also be increased by providing for success opportunities (SO) that are meaningful, are challenging, bolster achievement, and avoid boredom. Lastly, to improve confidence, Keller advocates a sense of personal control (PC) where the learner is allowed as much control of the learning experience as possible. Following Keller’s (1999a, 1999b) advice, the confidence-enhancing tactics were designed to be appropriate for the audience, the delivery system and the course, to be in line with course objectives and assessments, to be integrated with instruction (provide a minimal level of disruption to the learning process), to be cost-effective, and to fit within the time constraints of the class.
Participants
The subjects in this study were undergraduate students enrolled at a Texas university rated Carnegie Doctoral/Research Universities—Extensive. Subjects were enrolled in multiple sections of an online, freshman-level, for-credit computer course. All enrolled subjects in all sections were combined into a single pool and then were selected for the treatment or control group using a table of random numbers matched to the last four digits of their student IDs. They were then assigned to either the treatment or the control group sections in WebCT and SAM Office 2003.
The initial sample consisted of 81 (treatment n = 41; control n = 40) total students and included 37 males (treatment n = 18; control n = 19) and 44 females (treatment n = 23; control n = 21). Ages ranged from 18 to 31. Student-reported ethnicities were in line with university-reported demographics concerning the campus undergraduate population as a whole.
Confidence Tactics (CT)
| CT Components | Treatment Group | Control |
|---|---|---|
| LR1: Are there clear statements, in terms of observable behaviors, of what is expected of the learners? | Objectives were stated in SAM at the beginning of each lesson and restated on guide-sheets. Reminders were stated in the confidence-enhancing e-mails (CEE). In addition, a performance exercise (see SO1) served to familiarize learners with what was expected of them. | Objectives were not stated, and a pretest was not provided. |
| LR2: Is there a means for learners to write their own goals or objectives? | SAM 2003 is a self-contained simulation environment, so this was not an option. | SAM 2003 is a self-contained simulation environment, so this was not an option. |
| SO1: Multiple entry points: Provide a pretest and multiple entry points into the instructional material. | The treatment group received a performance exercise that determined the level of expertise the learner brought to each exercise, and this allowed for the learner to enter the training/instructional material at differing points. Each learner received training/instructional materials only in areas of demonstrated deficiency. Learners were reminded of this in the CEEs. | The control group received no such pretest/performance exercise and was required to take all of the training/instructional material regardless of previous knowledge, experience or expertise. |
| SO2: Is the content organized in a clear, easy-to-follow sequence? | The content was organized in a pretest-training-posttest sequence. The treatment group received a statement with each lesson assuring them the material was clear and easy-to-follow along with directions highlighting how to proceed through the pretest-training-posttest sequence. Learners were reminded of this in the CEEs. | This group received no such explanation. |
| SO3: Are the tasks sequenced from simple to difficult within the material? | Materials in SAM 2003 follow a logical sequence and are generally sequenced from easy to more difficult in each lesson. However, only the treatment group received a statement assuring them of this fact. Learners were reminded of this in the CEEs. | The tasks were sequenced from simple to difficult; however, the control group received no statement. |
| SO4: Is the overall challenge level appropriate for this audience? | Yes, but only in the treatment group was this stated to the learner. Learners were reminded of this in the CEEs. | Yes, but not stated. |
| SO5: Are the materials free of “trick” or excessively difficult questions or exercises? | Yes, but only this version stated this fact to the learner, and learners were reminded of this in the CEEs. | Yes, but this fact was not stated. |
| SO6: Are the exercises consistent with the objectives? | Yes, however, only this version stated the objectives to the learner before beginning. Learners were also reminded of this in the CEEs. | Yes, but objectives were not stated. |
| SO7: Are there methods for self-evaluation? | Yes, SAM 2003 was set to display simple feedback for each task (e.g., correct or incorrect). Results were also displayed at the end of each exam as a percentage (e.g., 90% correct). Learners were reminded of this in the CEEs. | No feedback was provided, and no results were displayed. |
| PC1: Are learners given choices in sequencing? Can they sequence their study of different parts of the material? | All exercises in each module were presented at once, and learners were able to approach the lessons in any order they chose. Learners were reminded of this in the CEEs. | Learners were given the lessons in a particular sequence, one-at-a-time, with a specific due date. |
| PC2: Are learners allowed to go at their own pace? | Self-pacing was allowed with a due date established clearly up front, and all assignments were opened at the same time and stayed open until the due date with no time-limits for self-pacing. Learners were reminded of this in the CEEs. | Each exercise was timed. The time-limit was decided as follows: (a) examine the time it took for the students in the previous semester to complete exercises, (b) select the longest time for completion, and (c) add thirty minutes. The control group had ample time to complete the exercises but was not informed of this. Every control group subject finished each exercise before time had expired. |
| PC3: Are learners given opportunities to create their own exercises or methods of demonstrating competency? | Learners were given the opportunity for demonstrating further competency by creating their own exercises (such as an Access database) for extra credit or to take the place of a low test score. Learners were reminded of this in the CEEs. | Learners were given no such opportunity. |
| PC4: Are learners given choice over study location? | Yes—this was an internet-based class. Learners were reminded of this in the CEEs. | Yes—this was an internet-based class. |
| PC5: Are learners given the opportunity to record comments on how the materials could be made more interesting? | A blog and threaded discussion concerning the materials was set up to allow for comments. Learners were encouraged to participate in the CEEs. | There was no access to a blog or threaded discussion about materials. |
| PC6: Are learners given the opportunity for feedback and practice in a “low risk” environment where it is acceptable to make mistakes and learn from them? | On the pretest, training, and posttest, learners were given feedback regarding performance and were allowed multiple attempts at the posttest. They were reminded about these multiple attempts at the beginning of each exercise and in the CEEs. Only the first attempt was used to gather data to measure and compare performance. | The control group received no pretest, one timed attempt at the training with minimal computer-generated feedback, and one attempt at the posttest with no feedback concerning final performance. |
Variables
The independent variable (treatment) consisted of ARCS confidence tactics (see Table
1) distributed through SAM Office 2003 and
through confidence-enhancing e-mail messages in the WebCT environment (see Figure 1). The two main dependent variables under investigation were confidence and academic performance.
Instruments
The instruments used in the study were the Instructional Materials Motivation Survey (IMMS) survey and a posttest, and both were delivered in a Web-based format at a distance. Both the IMMS survey and the posttest were delivered immediately following the treatment around the middle of the semester. Created by Keller (1993), the IMMS was used to gauge the motivational effect of instructional materials. It was designed to assess the four components of the ARCS model, as well as an overall motivation score. In this case, an individual measure of learner confidence was highlighted. In this study, the IMMS was found to have a total reliability alpha of .93 based on the obtained scores. The reliability alpha for computed scores of the individual subsection of confidence in this study was .85.
Academic performance was measured using an identical posttest automatically generated by SAM Office 2003 after students completed the training/instructional materials. The posttest consisted of an interactive performance-based simulation that tested what the students had learned concerning Microsoft Access. Given the nature of the posttest and the propriety restrictions of the software, a copy of the outcome measure cannot be provided. However, a sample screen shot from the posttest is included (see Figure 2). For this study, the posttest was found to have a total reliability alpha of .86 based on obtained scores.
Screen Shot from Posttest Measure (reproduced with permission from Course Technology)
Screen Shot from Posttest Measure (reproduced with permission from Course Technology)
Research Design
This study used a true experimental, post-test-only, control-group design, and was undertaken using quantitative methods. Given the nature of this study, adopting a pretest/ posttest model was inappropriate. In this study, an initial performance exercise (similar in concept to a pretest) was used as a confidence treatment. There were also concerns about potential test effects and the short delay between a complimentary pretest and posttest; this led to the decision to use a posttest-only design (Gall, Gall, & Borg, 2003).
The control group received none of the confidence-building tactics. The treatment group received confidence tactics (see Table 1) through SAM Office 2003 and through confidence-enhancing e-mails in WebCT (see Figure 1). Participation in this study was voluntary.
Treatment was provided in four steps:
The instructor selected SAM Office 2003’s simulation of Microsoft Access to be used for the duration of this experiment and WebCT for the delivery of confidence-enhancing emails (CEE).
As outlined in Table 1, the instructor modified SAM Office 2003’s Access simulation by incorporating the interventions and tactics based on the component of confidence in Keller’s ARCS model for the treatment group.
The instructor composed supplementary CEEs (see Figure 1) to help disseminate the remaining confidence-enhancing tactics based on Keller’s ARCS model for the treatment group.
The instructor presented the materials, with and without modification, to the respective treatment and control groups.
The attention, relevance, and satisfaction components of the ARCS model were not intentionally incorporated into the design of this study in order to better isolate the variable of confidence in question.
Distance Education Software
SAM (Skill Assessment Manager) provides training scenarios for Microsoft Office in a lifelike, simulated environment designed to replicate Microsoft Office 2003. In the case of this study, the students were trained to use the Access database program for about 5-and-one-half weeks.
Three semesters of prior surveys indicated that Access was the Microsoft Office program with which students were least familiar. Therefore, Access was chosen to help control for any variance in student ability. The SAM software is widely distributed to universities across the country and claims to have served hundreds of thousands of students and educators since its inception in 1998 (Course Technology, 2005). WebCT, along with SAM, is another widely used distance learning application.
Results
Research Question 1: Will the confidence tactics used in this study produce statistically significant differences between the control group and the treatment group in terms of learner confidence as measured by the IMMS?
The results indicated there was not a statistically significant difference between the treatment and control groups for confidence as measured by the IMMS (p=.080). However, the reported effect size (d = .41) and estimated power (.53) at alpha .05 cannot be dismissed as insignificant. Further study is warranted before a definitive conclusion can be drawn. The means, standard deviations, skewness and kurtosis values, effect size, and approximate power for the confidence variable are reported in Table 2.
Research Question 2: Will the confidence tactics used in this study produce statistically significant differences between the control group and the treatment group in terms of learner performance as based on posttest scores automatically generated in SAM Office 2003?
Results for the Confidence Subsection of the IMMS
| Section | N | Mean | SD | Skewness | Kurtosis | p | Effect Size (d) | Approx. Power (p = .05) | Approx. Power (p = .01) |
|---|---|---|---|---|---|---|---|---|---|
| Treatment | 35 | 31.77 | 7.276 | −.607 | −.207 | .080 | .41 | .53 | .27 |
| Control | 37 | 28.70 | 7.356 | −.159 | −.641 | ||||
| Total | 72 | 30.19 | 7.428 | −.348 | −.619 |
Results for Posttest Measure
| Section | N | Mean | SD | Skewness | Kurtosis | p | Effect Size (d) | Approx. Power (p = .05) | Approx. Power (p = .01) |
|---|---|---|---|---|---|---|---|---|---|
| Treatment | 30 | 93.40 | 5.43189 | −.702 | .002 | < .001 | 1 | .98 | .91 |
| Control | 26 | 86.10 | 7.38568 | −.865 | 1.730 | ||||
| Total | 56 | 90.0 | 7.33922 | −.896 | 1.354 |
The posttest showed a statistically significant difference in performance between the treatment and control groups (p < .001) with a reported effect size of (d = 1) at alpha .05 for the posttest score. This can be interpreted to mean that the treatment group (n = 30), on average, scored approximately one standard deviation above the average mean of the control group (n = 26). Results are reported in Table 3.
Discussion
Students in the treatment group did not seem to find the designed tactics especially confidence-enhancing during the Access training. Similar to Moller’s (1993) findings, there seem to be at least three potential explanations: (a) The ARCS model is ineffective for improving learner confidence; (b) the confidence tactics and methods used in this study were implemented improperly or were somehow inappropriate for these subjects; and/or (c) the differences in confidence were too small to measure or were immeasurable with the instrument (IMMS).
Taking each of these possible explanations in turn, there are insufficient data to suggest that the ARCS model is somehow flawed or incomplete when it comes to addressing learner confidence. The model has shown an ability to increase learner confidence even when confidence was not the focus of the researchers’ investigations. The question is not whether the model, as a whole, can produce increases. It is whether the individual subsection of confidence can be targeted as a valid, independent construct that produces consistent results. One could argue that this study’s results suggest that independently targeting confidence for improvement may more difficult than the ARCS model would lead one to believe. This could potentially require a rethinking of the ARCS model as a series of interdependent (and not independent) constructs for improving motivation (Babe, 1995). Further study is warranted before conclusions can be drawn, but the results of this study continue to challenge researchers’ assumptions (Keller, 1987a; Naime-Diefenbach, 1991) that individual components of the ARCS model can be isolated for improvement.
The second possible explanation for the lack of a statistically significant difference in confidence is that the confidence tactics used in this study were ineffective or implemented improperly. This is a possibility. Through informal surveys, e-mails, and discussion board postings, some treatment group subjects indicated that they found some of the confidence tactics used in this study more effective than others—particularly those related to personal control. Researchers have linked increases in learner control to increases in confidence (and positive attitudes of learners) as well as decreases in learner anxiety (Bandura, 1977; Keller & Suzuki, 1988; Kinzie, 1990; Kinzie & Sullivan, 1989; Moller & Russell, 1994). However, some members of the control group indicated an appreciation for the strict structure, deadlines and pacing. Also, a majority of treatment group students (64%) who were given personal control to complete the assignments at any time during the 5-and-one-half-week window waited until the last 72 hours to “cram in” most of the assignments before they were due. Only 24% of the treatment group finished the required assignments before the last week. There is no real way of knowing how such procrastination affected the confidence levels of the treatment group, but one can imagine that procrastination brings with it an increase in learner anxiety. Anxiety has an inverse relationship to confidence, so the effect on confidence levels was probably not a positive one.
In this study, a comprehensive approach to improving confidence was taken where several tactics were applied in one treatment. As a result, any differences in the study (or lack thereof) could be the result of effects of any of the individual manipulations (or combinations thereof), and there is no clear way of determining to what extent each tactic may have contributed to any differences found. However, this was not the goal of the study. While it would be ideal to conduct dozens of studies, each isolating only one confidence tactic, this is not practical. And, given the nature of new distance learning software packages (which often allow for extensive customization of the learning environment), such a multifaceted “approach” to improving confidence may have its place. In retrospect, it would have also been potentially beneficial to use an instrument to get a baseline measure of learner confidence before applying the treatment. Getting a read on how confident learners were before beginning instruction would have helped more clearly explain any changes in motivation. Such a measure is recommended for future studies.
Third, another way of stating differences may be too small to measure is to say perhaps the IMMS survey is not sensitive enough to detect short-term changes. While no current research indicates that the IMMS survey is a poor or weak measure of learner confidence, it seems possible that this survey may not be sensitive enough to detect short-term changes. Perhaps the confidence enhancements are producing a desired effect, but the survey cannot consistently detect the changes over the short-term. Given the protean motivational nature of learners over time, the survey would need to be highly sensitive or delivered at precisely the right time to accurately reflect learner changes in confidence. Over a brief period, learners may not even be aware enough of a change to report it accurately. Again, further study is warranted.
Performance
The data showed that the students in the treatment group outperformed the control group on the posttest measure for this particular study. This is in line with previous research findings that suggest increases in motivation can translate into increases in performance or achievement (Bickford, 1989; Gabrielle, 2003; Song & Keller, 2001). Because of the multiple tactics applied in the treatment, specific explanations of changes are as difficult to pinpoint for performance as for confidence. One cannot say with certainty that any particular aspect of the treatment was effective. It might be more appropriate to say that this multiple-tactic approach to increasing performance may have merit and is worth further study.
Additional Analysis
It seems confounding that, for this study, the confidence tactics did not produce a noted increase in learner confidence but did seem to have an effect on performance. In an attempt to further explain these findings, analysis of the individual subsections of the IMMS was conducted. For this study, there were differences noted for attention (p = .015, d = .57), relevance (p = .001, d = .75), satisfaction (p = .002, d = .72), and overall motivation (p = .002, d = .72), but not confidence (p = .080, d = .41). The reliability alphas for computed scores of the individual subsections in this study were as follows: attention (.86), relevance (.80), satisfaction (.86), and overall motivation (.92).
One particular reason for the noted increases in A, R, S, and total motivation in this study may be an overlap of the confidence tactics and confidence-enhancing e-mails into the attention, relevance and satisfaction components. For instance, providing the treatment group the opportunity to create their own exercises or methods of demonstrating competency and allowing the treatment group access to a blog and threaded discussion for comments may have enhanced attention or even relevance. Tactics such as these might stimulate the learner’s curiosity to think of ideas for improvement that increase feelings of “connectedness,” or relevance, to the material.
Simply receiving the e-mails might serve to gain learner attention. The concern, verbal praise, and goal reminders expressed in the messages may have served to increase learner satisfaction and improved their sense of connectedness (relevance) to the subject matter. Again, this study highlights some of the challenges faced in trying to isolate confidence for independent enhancement.
Perhaps the most important finding in this additional analysis was that overall motivation was enhanced in learners through the application of external factors. That was the belief which initially guided this study and, despite any disagreement about the validity of the independent components of the ARCS model, the model, as a whole, was apparently effective in this study for increasing overall learner motivation and performance.
Conclusion
It appears that this comprehensive approach to using the ARCS model shows a possibility for addressing some of the motivational needs and performance concerns of online students. Although this initial study should not be generalized beyond undergraduates enrolled in the online, entry-level computer application course using the SAM Office 2003 software at the university in this study, the apparent positive performance and overall motivation results should encourage continued study.
