Course management systems (CMSs) support thousands of courses at colleges and universities worldwide, delivering fully online courses as well as supplementing traditional face-to-face instruction. While quite a few studies have focused on the numerous benefits and technological advantages of CMSs, there is a paucity of empirical work focused on user attitudes about the efficiency and ease of use of CMSs, and perceived usefulness of potential additional features. We developed an attitudinal scale to better define and quantify these constructs. We piloted the survey with 234 college students who used WebCT Vista as a supplement to a face-to-face course. In this article we describe the development and initial validation of the scale. Since there is so little empirical work in this area, validating the instrument and identifying attitude factors should contribute to future research in this fast-growing arena. We further discus findings from our quantitative and qualitative data that provide evidence that CMSs are still evolving tools, and need accommodations and improvements.
Course management systems (CMSs) such as WebCT, Blackboard, Angel, Educator, and FirstClass are software systems designed to manage course content and course activities. These tools integrate technological and pedagogical features into a well-developed Web-based system that allows instructors who are unfamiliar with Web-based technologies to design, deliver, and manage an online course. Common features in CMSs include content areas, discussion boards, chat rooms, assignment drop boxes, quizzes and surveys, and white boards. CMSs support student-to-student and student-to-teacher communication and collaboration. Students are able to share resources, collaborate, participate in forums, take online tests, access their grades, and upload assignments.
Today CMSs support thousands of courses at colleges and universities, and that number is growing at a staggering rate. While CMSs were initially developed to support distance education and fully online course delivery, they are also extensively used in on-campus classroom settings to compliment traditional face-to-face courses, or a so-called “blended” approach (Morgan, 2003; Simonson, 2007). The ease with which users can organize asynchronous and synchronous communication activities in CMSs is one of its most powerful features, because it enables (in fact, arguably encourages) instructors to create and support dynamic learning communities, consistent with a social constructivist perspective. This certainly explains some of the growth in oncampus blended courses (Dabbagh, 2004; Morgan, 2003).
“Perhaps no other innovation in higher education has resulted in such rapid and widespread use as the CMS” (Harrington, Gordon, & Schibik, 2004, “History of CMS” section). In 2002 over three-quarters of all colleges and universities in the US had adopted a CMS, and nearly one-fifth of college courses used a CMS (Campus Computing Project, 2002). By 2004, just a few short years from their introduction into higher education, such systems could be considered “ubiquitous” on college campuses (Molenda & Bichelmeyer, 2005). The recent merger of Blackboard and WebCT made Blackboard, Inc. the world’s leading provider of integrated e-learning systems, by itself serving more than 3,650 academic clients in more than 60 countries worldwide (Blackboard.com, 2005; Bradford, Porciello, Balkon, & Backus, 2007). It is possible that CMSs have introduced so much new functionality, so fast, that end users have not had a chance to seriously reflect on what they need or want; and have not yet critically examined these tools to ask questions like What else do I need? or How can these tools get even better to satisfy my educational expectations and needs?” (Ioannou & Hannafin, 2008).
West, Waddoups, and Graham (2007) interviewed 30 CMS-using university faculty and surveyed 122 more to determine adoption patterns and to identify obstacles to CMS adoption. The authors noted though that there is limited research available on implementation issues surrounding CMSs. While there are numerous studies focusing on the benefits and technological advantages of CMSs, which we certainly appreciate, studies that focus exclusively on the technological limitations and deficiencies of CMSs are almost non-existent. There is a paucity of empirical work focused on user attitudes about the efficiency and ease of use of CMSs, and perceived usefulness of potential additional features. In this study we attempted to develop a scale to better define and quantify these constructs. We conducted an initial validation of the scale and extracted and identified attitude factors. Then, using student responses to both a Likert-type attitude survey and to an open-ended question, we attempted to confirm user concerns reported in the literature about CMSs. We also tried to determine how much users value specific features/ functionalities that are currently unavailable in most CMSs. Identifying important features that users need in CMSs has obvious design implications. In addition, validating the instrument and identifying attitude factors should contribute to future research in this fast-growing area.
Limitations of CMSs
As CMS use has increased, instructors and students have begun to express frustration over performance and design features. Siemens (2004) examined the drawbacks of CMSs and suggested functionalities that CMSs need to acquire to meet user needs. The author asserted that the CMS interface is confusing to many users, and needs to be simplified. Current CMS interfaces rely on “what the designers/ administrators want/need to do, [rather than on] what the end user want/need to do” (Siemens, 2004, “Drawbacks to Learning Management Systems” section). In a recent study, Jafari, Mcgee, and Carmean (2006) investigated the advantages and limitations of CMSs, interviewing 59 faculty, 52 students, and 51 academic, library, and IT administrators, and IT service providers from seven campuses. These institutions were using one or more CMS, including Blackboard, WebCT, eCollege, and ANGEL Learning. All users expressed dissatisfaction with the speed, efficiency, and intuitiveness of current CMSs. Administrators suggested that “Smart error messages would relieve frustration and time for both faculty and support staff. Smart metatagging and searching could make accessing files and finding threads more efficient” (p. 4). Faculty members wanted more user-friendly functions that required fewer clicks in order to make tasks such as entering grades or returning files less time-consuming. They also indicated a desire for more feedback, warning, and notification mechanisms within the system. However, the most frustrated end users were the learners who generally characterized the current CMSs as “dull.” The authors observed:
For all the slow, backward, and clunkiness of the current systems, students noted that the environment isn’t even easy to figure out: the navigation is confusing and takes too many clicks, assignments get lost, the discussion can’t be sorted with newest on top, the system doesn’t learn anything about the learner and never sends reminders or status messages. (p. 8)
Considering how technology’s limitations affect the quality of the instruction, Ioannou and Hannafin (2008) discussed how CMSs can become more efficient, user-friendly, and intuitive by adopting client-site software. The authors questioned how much users value the many features that vendors include in CMS product updates. Instead, they argued, vendors may be better served investing in product enhancements that improve speed and ease of use, two things users care about and that are both characteristics of a client application. They argued that CMS client-site software could offer for example, faster response time, allow users to work off-line and synchronize, enable real-time spell checker and dynamic notification for changes in the online course, allow changes in the look-and-feel, and provide for set-up of a personalized keyboard-driven interface. The authors concluded that CMSs have technological limitations and need accommodations to satisfy educational expectations.
Finally, in terms of instrumentation, we failed to find any work on the development of valid and reliable scales in this area. Thus, developing and validating such a scale should be of interest to researchers and developers of CMSs.
Method
A total of 234 college students from a large public university in the northeast participated in this pilot study. These students used WebCT Vista (Vista) in a C++ programming course in Fall 2006. Most of the students were freshmen (N = 198), White (82%), males (N=200), and majored in various engineering programs (civil, biomedical, chemical, computer science, electrical, mechanical). They completed our online anonymous scale during the last week of classes. The instructor, who had been using WebCT extensively in this course for several semesters, was very interested in the results of this research as a means to improve usability and overall course design.
The Scale
We interviewed the course instructor regarding her attitudes and opinions about using Vista for three semesters in this course (including the semester we conducted this study). Based on her feedback, our review of the literature, and our own experiences using Vista both as a student and as an instructor, we developed a scale dealing with user attitudes towards Vista (see Appendix A). The scale included general questions about the Vista use, such as: Have you used Vista in other classes? On average, for how long did you work in Vista for this course per week? What WebCT Vista tools did you use for this course? To access Vista what computer did you mostly use? In addition, we designed 11 items to assess student satisfaction with Vista, on a 5-point Likert response scale (from 1 = strongly disagree to 5 = strongly agree). Another eight items were designed to assess students’ perceived usefulness of specific new features currently unavailable in Vista, also on a 5-point Likert scale (from 1 = not at all useful to 5 = extremely useful). Two content experts were asked to review the scale separately. Based on their feedback, the scale was revised and finalized.
Procedure
During the semester, the instructor employed a blended learning approach, using Vista to supplement the classroom instruction. According to the instructor and one of the course teaching assistants (who were interviewed), Vista was used for sharing instructional materials (PowerPoint slides, sample code/exams, assignments/exams solutions), submitting assignments, posting questions/discussions, distributing grades and exam statistics, and taking practice quizzes.
The vast majority of students (97%) reported that they had used Vista in other classes. Approximately 75% reported that they accessed Vista mostly using their personal computers/laptops. More than 70% of students reported that, for this specific course, they logged into Vista at least 4 times a week. Approximately 55% reported that they worked in Vista for one to four hours a week; 20% reported working for less than one hour, while another 15% reported working for more than four hours a week. Based on these responses, and also from our discussion with the instructor and teaching assistant, we concluded that Vista was extensively used in this programming class.
Results
Scale Design
As mentioned earlier, one objective of the study was to lay a foundation for future research by developing a valid and reliable instrument. We carried out an exploratory factor analysis to determine the factor structure of the data, and to decide if any items should be eliminated from the final version of the scale. A Principle Axes Factor (PAF) analysis with Oblimin rotation, conducted on the 19 Likert-type items in the scale, extracted five factors that had eigenvalues greater than 1.0. PAF analysis with Varimax rotation suggested the same factor structure. The five factors explained 59.0% of the total variance in the items. Inspection of the pattern coefficients showed that three items loaded highly on more than one factor and therefore were excluded from the final solution.
The first factor extracted by the PAF analysis seemed to measure user-satisfaction and was composed of five items including:
My overall experience with the course was improved;
My anxiety was reduced because I knew where to find class material;
It saved me time
Vista is a neat tool; and
I would like to use Vista in more classes.
The second and third factors, Corrective Feedback and Personalized Environment, both measured the extent to which new (currently unavailable) features would be useful to these WebCT Vista users. The Corrective Feedback factor included items:
Error warning messages;
Undo and redo functions; and
Real-time spell checker with wavy red underlines.
The Personalized Environment factor included:
Customizable look and feel;
Ability to change the organization of the screen; and
Keyboard-driven interface that does not require the mouse.
Cronbach’s Alphas Reliabilities (N = 234)
| Scale | Alpha |
|---|---|
| User satisfaction | .79 |
| Corrective feedback | .75 |
| Personalized environment | .72 |
| Response time | .67 |
| Ease of use | .45 |
| Scale | Alpha |
|---|---|
| User satisfaction | .79 |
| Corrective feedback | .75 |
| Personalized environment | .72 |
| Response time | .67 |
| Ease of use | .45 |
Frequency of Use of Several Vista Tools (N = 234)
| Tool Name | Frequency | Percent |
|---|---|---|
| Calendar | 85 | 36.3 |
| Bookmarks | 5 | 2.1 |
| Discussion forums | 36 | 15.4 |
| Internal email | 23 | 9.8 |
| Online journals/notes | 38 | 16.2 |
| Progress review | 88 | 37.6 |
| Real-time chat | 11 | 4.7 |
| Submit assignments | 225 | 96.2 |
| Whiteboard | 2 | .9 |
| Tool Name | Frequency | Percent |
|---|---|---|
| Calendar | 85 | 36.3 |
| Bookmarks | 5 | 2.1 |
| Discussion forums | 36 | 15.4 |
| Internal email | 23 | 9.8 |
| Online journals/notes | 38 | 16.2 |
| Progress review | 88 | 37.6 |
| Real-time chat | 11 | 4.7 |
| Submit assignments | 225 | 96.2 |
| Whiteboard | 2 | .9 |
The fourth factor was composed of items “I am satisfied with the system response time” and “The time required to load material was reasonable.” This factor, named Response Time, measured how satisfied students were with the speed in Vista. The fifth and last factor, Ease of Use, was composed of items “Navigation was confusing” and “It was easy to figure out the functionalities.”
Following the factor analysis, Cronbach’s alpha reliability analysis was conducted on each factor (see Table 1). A widely accepted and established level of adequacy for Cronbach’s alpha is .70 (Netemeyer, Bearden, & Sharma, 2003), indicating that caution needs to be taken in interpreting the Response Time and Ease of Use subscales. Further, Clark and Watson (1995) advised against subscales that include only two items, recommending that each factor within a scale have at least four or five items for very narrow constructs. We include all scales given the exploratory nature of this investigation but advise caution and concede that further testing is needed.
Student Use of Vista Features
Despite extensive use of the Vista site, the frequency of use of several functions in Vista, as reported by the students, was limited (see Table 2). This confirms, in part, the argument by Ioannou and Hannafin (2008) that often users do not look for, or perhaps are not aware of, the many features that vendors seek to implement in CMSs.
Student Means and Standard Deviations for Subscales (N = 234)
| Subscale | # Items | Mean | Standard Deviation |
|---|---|---|---|
| 1. User satisfaction | 5 | 3.84 | .65 |
| 2. Corrective feedback | 3 | 3.32 | .93 |
| 3. Personalized environment | 3 | 2.58 | .94 |
| 4. Response time | 2 | 3.67 | .84 |
| 5. Ease | 2 | 3.59 | .73 |
| Subscale | # Items | Mean | Standard Deviation |
|---|---|---|---|
| 1. User satisfaction | 5 | 3.84 | .65 |
| 2. Corrective feedback | 3 | 3.32 | .93 |
| 3. Personalized environment | 3 | 2.58 | .94 |
| 4. Response time | 2 | 3.67 | .84 |
| 5. Ease | 2 | 3.59 | .73 |
Note: For subscales 1, 4,and 5, the response scale ranged from 1 (strongly disagree) to 5 (strongly agree). For subscales 2 and 3 the response scale ranged from 1 (not at all useful) to 5 (extremely useful).
The mean factor scores for student responses are displayed in Table 3. The mean of 3.84 (out of 5) for the user-satisfaction subscale suggested that students were generally satisfied with WebCT Vista. Paradoxically, however, when users were asked to rate the usefulness of new features that are currently unavailable in Vista, they tended to rate those missing features as equally useful. Specifically, 90% of the students reported that having “undo” and “redo” functions in Vista would be at least useful (3 or more on a 5-point Likert scale). More than 82% of the students reported that having “electronic notification for changes in the course, arriving in the form of pop-up window or sound” would also be at least useful. The majority of the students reported that functions such as real-time spell checker with wavy red underlines (78%), error warning messages (78%), and ability to change the organization of the screen (60%) would be at least useful.
The open-ended responses also seemed to support the idea that students have unmet needs, despite the overall positive satisfaction rating. When asked whether they ever learned after the fact that they had missed something on the Vista course area, 37 students (16%) responded positively. In a follow-up question asking students to explain what they had missed, the responses seemed to be related to the need for dynamic notification for changes in the online course, as discussed by Ioannou and Hannafin (2008). Specifically, students reported missing new assignments posted, changes in the instructional material posted, changes in dues dates, changes in the calendar/ announcements, and important discussion postings. Sample responses include: “I missed changes in assignments due dates”; “I was unaware that a new online exam had been posted”; “I've missed things on discussion boards, I never knew when something new was posted”; “When some of the assignments were modified slightly I missed it, as well as hints about the homework”; “I didn't look at the discussion boards when there were hints about the homework/lab assignments”; “I missed announcements about classroom change.”
Only nine students made negative comments regarding submitting their assignment on Vista. Related to the issues raised by Jafaris et al. (2006) about lack of feedback, warning, and notification mechanisms within the system, five students reported that although they thought that they had submitted their assignments successfully, they found later that they had not (e.g., “I thought I submitted a homework assignment, but then a few days later the computer said I did not submit it”). Four students complained that they failed to submit an assignment (or the correct assignment), because the uploading procedure was confusing (e.g., “Uploading assignments can get confusing, which led to sending the wrong assignment”). This is consistent with Siemens’ (2004) discussion about CMSs having confusing interfaces.
Discussion and Future Directions
Review of the literature revealed that we haven’t yet systematically measured user attitudes about CMSs. In this study, we attempted to quantify college students’ satisfaction with WebCT Vista, and to assess the perceived usefulness of specific new features currently unavailable in Vista. Five factors were identified: user-satisfaction, corrective-feedback scale, personalized environment, response time, and ease of use. These factors are consistent with previous discussions in the literature about the deficiencies of CMSs. Validating the instrument and identifying attitude factors should contribute to future research, especially related to the design of a new generation of CMSs.
A few limitations to the study must be acknowledged. First, the sample was very homogeneous in terms of age, gender, and race. Further, the majority of the students were majoring in engineering. It is quite likely that members of this sample share attitudes about many issues, including CMSs. And courses in other disciplines would use the tool and value different CMS features. We are confident that the procedures we used to develop the factors and the scale are still valid. However, we readily acknowledge that the attitudes of this particular group are not representative of college students in general and encourage more work with heterogeneous samples in this area.
Future studies should focus on the further development of reliable and valid scales. Two of our subscales did not have acceptable internal consistency. Thompson (2004) explained that poor score reliability may compromise the ability of the scores to measure intended constructs. In the future, we should improve the scale, based on the 5-factor structure we obtained. More items need to be developed for all the subscales to not only improve their reliability, but to also improve the content coverage (validity) of the construct. Once a good measure of the constructs is finalized, future studies could explore issues such as how CMSs can be improved or adapted to suit the needs of users with different characteristics and learning needs.
Our findings based on the quantitative and qualitative analysis provide evidence that CMSs are still evolving. Products such as WebCT Vista have weaknesses that need to be addressed. Future studies need to focus on the technological drawbacks of CMSs, because technology’s affordances and limitations highly affect the quality of the instruction (Kozma, 1994). At the same time, as users become more familiar with CMSs, they will almost certainly more critically evaluate the tools and demand accommodations and improvements to improve their online experiences, and to make learning with CMSs more efficient and enjoyable.
Appendix
1. Have you used WebCT Vista in other classes in addition to CSE 123?
❒ Yes
❒ No
2. On average, how many times per week did you log into WebCT Vista for CSE 123 (choose one)?
❒ Less than once
❒ Once
❒ 2-3 times
❒ 4-5 times
❒ 5-7 times
❒ More than 7 times
3. On average, for how long did you work in WebCT Vista for CSE 123 per week (choose one)?
❒ Less than 30 minutes
❒ 30 minutes - 1 hour
❒ 1-2 hours
❒ 2-3 hours
❒ 3-4 hours
❒ More than 4 hours
4. To access WebCT Vista what computer did you mostly use (choose one)?
❒ My personal computer / laptop
❒ A computer in the labs
❒ A computer at the library
❒ A computer at work
❒ Other, please specify ___________________________________
5. What WebCT Vista tools did you use for CSE 123 (select all that apply)?
❒ Calendar
❒ Bookmarks
❒ Discussion forums
❒ Internal email
❒ Online journal/notes
❒ Progress review
❒ Real-time chat
❒ Searching within the course
❒ Submit assignments
❒ Whiteboard
❒ Other, please specify ___________________________________
6. Select outside tools that you have used to communicate with the instructor about the class (select all that apply).
❒ E-mail
❒ Instant messenger
❒ Face-to-face communication
❒ Other, please specify ___________________________________
7. Select outside tools that you have used to communicate with other students about the class (select all that apply).
❒ E-mail
❒ Instant messenger
❒ Face-to-face communication
❒ Other, please specify ___________________________________
8. Rate how much you agree with the following statements about using WebCT Vista in CSE 123:
|
|
My overall experience with the course improved. My anxiety was reduced because I knew where to find class material. It saved me time. The number of clicks to complete a task was reasonable. I am satisfied with the system speed/ response time. Navigation was confusing. It was easy to figure out the functionalities (e.g., calendar, submit assignments). The time required to load material was reasonable. The interface was simple/ user-friendly. WebCT vista is a neat tool. I would like to use WebCT Vista in more classes. | ❒ Strongly disagree ❒ Disagree ❒ Neither Agree or Disagree ❒ Agree ❒ Strongly agree |
9. Did you ever learn after the fact that you missed something on the WebCT Vista course area (e.g., a change in assignment due date, a hint about an assignment posted on the discussion board)?
❒ Yes
❒ No
10. If yes, please explain what you missed?
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
11. How often did you use the following material?
|
|
Assignment solutions Computer account information Chapter goals Exam solutions Exam statistics FAQ information Glossary Grading sheets Help information PowerPoint slides Sample programs/code Sample exams Software | ❒ Almost never ❒ Once in a while ❒ Sometimes ❒ Often ❒ Almost all the time |
12. List other resources or materials you would have liked to have available for CSE 123?
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
13. Currently the following features are not part of WebCT Vista. How useful do you think they would be?
|
|
Work off-line, without being connected to the internet. Customizable look and feel of the environment (e.g., fonts, icons). Electronic notification for changes in the course (e.g., new calendar posting), arriving in the form pop-up window like in instant messenger, or sound like in e-mail. Error warning messages (e.g., Do you want to save this file?). “Undo” and “redo” functions. Real-time spell checker with wavy red underlines. Keyboard-driven interface that does not require the mouse (e.g. set ctrl+D to open your calendar). Ability to change the organization of the screen (interface). | ❒ Not at all useful ❒ Somewhat useful ❒ Useful ❒ Very useful ❒ Extremely useful |
Demographics:
14. Year of study
❒ Freshman
❒ Sophomore
❒ Junior
❒ Senior
15. Major:
___________________________________________________________________________
16. Gender
❒ Male
❒ Female
17. Ethnicity (optional)
❒ African American
❒ American Indian or Alaskan native
❒ Asian /Pacific Islander
❒ Hispanic American
❒ White/Caucasian
