EVALUATION OF A WEB-BASED MASTER’S DEGREE PROGRAM: Lessons Learned From an Online Instructional Design and Technology Program

Evaluators who study distance education programs generally identify several key factors (aspects or areas) of a program to evaluate. For instance, based on stakeholders’ concerns, Levin, Levin, Buell, and Waddoups (2002) identified eight focus areas for evaluating an online master’s program at the University of Illinois: interface design, instructional design, student satisfaction, faculty satisfaction, economic viability, departmental capacity, interdepartmental collaboration, and college level infrastructure. Likewise, Van Slyke, Kittner, and Belanger (1998) created four categories for use when evaluating distance education programs, including: institutional characteristics, learner characteristics, course characteristics, and distance learning characteristics (cited in Belanger & Jordan, 2000). Mauldin (2001) identified the critical characteristics of a distance education program as: student characteristics and practices, faculty characteristics and practices, curricula design, technology, and organizational support.

At present, the most popular program evaluation models are objective-oriented evaluation, management-oriented evaluation, participant-oriented evaluation, customer-oriented evaluation, and expertise-oriented evaluation (Broadbent & Cotter, 2003; Fitzpatrick et al., 2004). These evaluation models are used to evaluate programs from different perspectives. For example, the objective-oriented evaluation model tends to answer such questions as whether and to what extent an intended program meets its objective(s). The expertise-oriented evaluation model answers such question as what experts think about whether and to what degree the evaluated program is successful. Selection of evaluation models is determined based on the purpose of the evaluation and the audience for the evaluation report (e.g., Broadbent & Cotter, 2003; Fitzpatrick et al., 2004).

The present evaluation is most closely aligned to the purposes of the participant-oriented and expertise-oriented evaluation approaches. Proponents of participant-oriented evaluation view participants as central to the evaluation. Using this approach, evaluators work to portray the multiple needs, values, and perspectives of the program stakeholders in order to make judgments about the value or worth of the program (Fitzpatrick et al., 2004). For this evaluation, the major participants (stakeholders) were identified as the program administrators, faculty, and students. This evaluation focused on how the major stakeholders perceive the quality of the DM program.

Following the purposes of expertise-oriented evaluation, the evaluators for this study also reviewed the literature about evaluation of online programs and referred to relevant standards in the field. Relevant standards that guided this evaluation include: Benchmarks for Success in Internet-based Distance Education (The Institute for Higher Education Policy [IHEP], 2000); North Central Association Commission on Institutions of Higher Education [NCACIHE], 2000); and Chickering and Gamson’s (1987) Seven Principles for Good Practice in Undergraduate Education. Key factors of successful online programs that were identified from the literature include: program management, course management, technology, quality assurance, faculty support, and student support (Flowers, 2000; Judd, 1998; Law, Hawkes, & Murphy, 2002; Mauldin, 2001; Sherry, 2003).

This evaluation involved a mixed-methods approach that incorporated data collected from interviews and an online survey. Participants of the study included administrators, faculty, and students of the DM program. Administrators were interviewed during March 2004, an online survey was administered to students in April 2004, and faculty members were interviewed during April 2004.

Five administrators were interviewed for this evaluation, including the department chair, program director, program coordinator, current program graduate assistant, and a past program graduate assistant. The five administrators were interviewed during face-to-face meetings, and the interviews were audiotaped.

The interview questions, drawn from the literature, included topics such as the mission and purposes of the DM program, the structure of the program, the methods for recruiting DM students and faculty, quality control, and the strengths and weakness of the programs. The interview questions were piloted prior to the administrator interviews with an administrator from an online degree program at a different university in order to test for clarity of the items.

Six faculty members were interviewed: two full-time (tenured) and four adjunct faculty members. All four adjunct faculty members are alumni of the department’s residential doctoral program. The full-time faculty members were interviewed face-to-face, and the adjunct faculty members were interviewed by telephone. The face-to-face interviews were audio taped.

The interview questions included such topics as the quality of learning outcomes for the program, support provided to students, support received from the program, and the design of courses. The faculty interview questions had been previously piloted with two faculty members from an online degree program at a different university to test for clarity.

After the administrator interviews were completed, an e-mail invitation was sent to the entire population of 77 students and alumni of the DM program. Nineteen students initially completed the survey. An e-mail reminder was sent 1 week later, which generated four more completed surveys for a total of 23 completed surveys. The return rate was 29.87% for the student surveys.

The anonymous online student survey consisted of 43 questions: 6 demographic questions, 7 open-ended questions, and 30 close-ended questions with a 5-point Likert response scale. The survey questions addressed topics such as course management, program management, technical aspects of the program, and overall quality.

Administrators identified four reasons for offering the DM program in Instructional Design and Technology. The first purpose of the DM program was to maintain and/or increase the reputation of the program in the field. Administrators explained that they felt in order to be reputable in the field of instructional design and technology, they needed to establish a presence in the area of online education and Web-based instruction.

The purpose of maintaining reputability is closely tied to the second reason identified by administrators for establishing the DM program, which was to provide research and development opportunities for faculty and PhD students in the department. In particular, the program director and department chair discussed the importance of providing opportunities for faculty and students in the field of IDT to engage in design, development, implementation, and research related to online learning and instruction in order to ensure that the program is current with the field.

The third purpose identified by administrators for offering the DM program was to provide educational opportunities for those who could not enroll in the residential program because of work or family responsibilities that make it impossible to move to a campus location. In essence, this purpose had to do with opening up the market for the masters program to reach students who would not be able to participate in such educational opportunities without the aid of online technologies.

The fourth reason identified by administrators for offering the DM program was to increase the revenue stream for the department. Though the department administrators were looking for ways to increase revenue, neither the program director nor the department chair were sure whether the DM program was turning a profit. The lack of certainty about the financial impact of the program is the result of the fact that, as an academic unit of a public university, the department has little control over its own financial situation. Additionally, program administrators emphasized that because the program is housed in a department of a public university, the main focus of public service would always take precedence over issues of profitability. This helped explain the fact that there was no formal business plan created when the DM program was started.

It is worth noting that two of the four reasons for starting the DM program had to do with meeting the needs of students (these reasons were opening the program to a new market and providing research and development opportunities for residential students), while the other two reasons for starting the DM program had to do with maintaining or enhancing the reputation, credibility, and health of the department, its existing programs and the work of the faculty. The overall rationale for the DM program identified by these administrators indicates that they believe, in the field of Instructional Design and Technology, online learning environments are potentially beneficial for the key stakeholder groups including students, faculty, and administrators.

The IDT DM program that was the focus of this evaluation used a cohort model for the first two student groups that were admitted in 2000 and 2001, and changed to a noncohort model in 2003. In the cohort model, students were required to participate in a preprogram residential orientation for 1 week at the university, and the students were enrolled in courses together as a cohort group. In the noncohort model, the 1-week residential orientation was eliminated and students were not required to take courses as part of a cohort group. Also, more adjunct faculty members have been employed in the program since the noncohort model was instituted in 2003.

The administrators who discussed the cohort model all emphasized that one advantage of having cohorts was to help build a sense of community. They all expressed concerns that distance students experience an inherent feeling of isolation when taking courses in an online format. The program coordinator recalled the preprogram residential orientation for the first two cohort groups. She

commented that the residential orientation for the first cohort group was well received, as it was held together with the orientation for traditional residential students, and conducted right before the fall semester started. On the other hand, she did not think the residential orientation for the second cohort went as well because the orientation was given during the summer session (early August) and many residential faculty and students were out of town. Without the support of faculty and residential students, the administrators were overwhelmed with the responsibilities of conducting the orientation.

One major reason the cohort model was discontinued, according to the program directors, was because of its rigid structure, which contradicted the flexible nature of the online courses that drew students to the DM program in the first place. Because the university campus is not located in a big city, travel and lodging were not convenient for DM students, and timing the orientation at the beginning of the fall semester was inconvenient because many of the DM students were K-12 teachers who were just beginning their own school year at the time of the orientation.

Four strengths of the program were identified by administrators. First, the quality of the DM program was considered equivalent to that of the residential program. According to the administrators, the program uses the same criteria in admitting and awarding degrees to DM students as for the residential program. No distinction is made between the DM program courses and the traditional residential master’s program courses; the same core characteristics and requirements are found in both. Neither the courses nor the degree are identified as being at a distance on the university transcripts.

Second, administrators believe that there is a reciprocity between the distance and residential programs that provides benefits for both programs in many areas. For instance, a number of residential students have helped design and teach the courses for the DM program, which has provided an important practicum experience for the residential IDT students and improves the quality of interactions for distance students. Additionally, the administrators pointed out that lessons learned during the design and development of the DM course materials has led to enhancements and improvements in the residential course offerings.

Third, the courses offered in the distance master’s program are project-based, teambased, practical experiences that are designed to help students develop a well-rounded set of skills in instructional design. Several administrators interviewed for this evaluation expressed the belief that the emphasis on pedagogical experience rather than technology was another strength of this DM program.

Finally, several administrators expressed the belief that using qualified adjunct faculty was a strength of this DM program. According to the program director, compared to the fulltime faculty, the adjunct faculty members were less expensive than tenured faculty and received high ratings from students in course evaluations. In contrast, however, the program coordinator mentioned that occasionally she received complaints about the teaching quality of the adjunct faculty.

Three key areas of the DM program were identified by administrators as needing improvement. First, technology was a big concern of the administrators who were interviewed for this evaluation. These administrators expressed the belief that technology hindered the DM program in the areas of registration, administration, and course management. The program director complained that the technology provided by the university did not provide a full range of capabilities needed for offering online programs. This created a dilemma: while the technology provided by the university was not effective, better technical tools were not supported by the university technology services.

Second, though the program had made attempts to address the problem that online students often feel isolated, several administrators expressed the feeling that this is still an area that needs to be improved. The dilemma for administrators is that, while they want students to feel a sense of belonging to the program, they recognize that distance students generally have full-time jobs and families and other responsibilities that keep these students from high levels of engagement beyond the work they do in their individual courses.

Last, administrators expressed the feeling that students need to be better informed about the program before they start so that they may have more appropriate expectations about the curriculum, instructional activities, and time required to complete the program. Students occasionally complain that the courses the program offers are inadequate to help them acquire expertise in the areas they expect. As one administrator explained, the DM students are from different backgrounds and join the program with varying expectations. For instance, students who are full-time school teachers expect to improve their knowledge and skills in the educational setting, while those from business settings, such as those working as corporate instructional designers, expect to learn more about training and design. As a result, while the program strives to meet the needs of all students by offering various courses, students from an educational setting want the program to minimize the business focus while focusing more on educational aspects. In contrast, students from the corporate environments want the program to increase the business focus while decreasing the focus on educational contexts.

The six faculty members who were interviewed for this evaluation expressed consistent themes in their responses to interview questions. The resulting comments have been cate gorized into three areas: teaching online, administration of the program, and technology issues.

The majority of comments from faculty about the experience of teaching in the online program focused on the benefits and drawbacks that teaching online has to offer both instructors and students. The primary benefit mentioned about teaching online was consistently the improved flexibility that teaching online offers an instructor. This flexibility was shown in the physical locations of the adjunct faculty, none of whom reside in the location where the DM program was hosted. The adjuncts noted that they would not be teaching for the DM program if it were not for the flexibility inherent in online teaching. Additionally, the lack of set classroom times allowed instructors and students a more flexible schedule.

Many of the faculty mentioned that the best feature of the distance master’s program was that it gave students who would not otherwise be able to, a chance to take courses from a well-respected program.

Several of the faculty, both adjunct and fulltime, mentioned that the online students were different than the residential students in that the vast majority of online students were fulltime employees and taking classes with a focus on using what they learn to help them in their current positions. Faculty members believe that this difference results in a stronger focus on the utility of the knowledge gained, but perhaps a shallower interest in knowledge for its own sake. Several faculty members noted that the online students usually did not explore topics as deeply as the residential students did.

Another advantage of the online program noted consistently by the faculty was that they believed the online classes are representative of the residential program. The online courses were created from the residential courses, and the faculty members believe that the distance courses present the same level of challenge to students as the residential courses.

While faculty noted that students interacting in an asynchronous environment could have more time to reflect on their answers and therefore provide higher quality comments, and the “faceless” format of writing e-mails and forum comments encourages shy students to participate more, faculty also noted substantial disadvantages to online interactions. Every faculty interviewee noted the primary disadvantage of teaching online is restricted communication, which can hinder the instructorstudent relationship, as well as communication between students in the course. In an online environment that is dependent upon textual communication, it is very easy to be misunderstood as there is no supplementary communication, such as body language. Therefore, misunderstandings among students working in groups may be more common.

The faculty interviewed for this evaluation consistently noted that the “feel” of interacting with students in an online environment was different from the relationships formed in a residential class in that instructors do not see their students’ faces or see the students at a set time each week. While several instructors noted the advantages of administering a course in the online environment in terms of tracking and storing student communication, a number also pointed out that students might feel less involved in an online environment, and it can be difficult to excite or motivate students in such a sterile setting.

Additionally, faculty felt that the emphasis on written communication in an online environment requires more effort in communication. Providing feedback, as well as communication in general, can be more difficult and time-consuming in online environments. Reading student e-mails and posts and then responding takes up a great deal of time, according to the faculty.

Finally, faculty noted that the format of online communication has inherent restrictions. There are some activities available in a residential course that simply cannot be effectively replicated online. Several faculty members noted the difficulty of trying to facilitate online class discussions through real-time chats, while others noted the lack of relationship among students, particularly as compared to the group work in the residential program. While the online program still uses groups, with outside restraints and a more sterile form of communication, some faculty felt that it is difficult to simulate the effectiveness of the residential program in this regard.

While most of the faculty comments focused on teaching online, some faculty discussion did address the administration of the distance master’s program. All faculty members who discussed the current technical support provided by graduate assistants and librarians in charge of the electronic reserves system were complimentary; however, the adjunct faculty noted there were problems caused by their lack of direct control over course resources. Some frustration was expressed that any adjustments or problems with the courses had to be directed to the graduate assistants or librarians, which makes administering the course more difficult.

Additionally, because adjunct faculty teach courses which they have not designed, some felt that they did not have the freedom to change their courses, while others simply noted that there was no incentive for them to do so, given their salaries, even though they felt that designs could be improved or the content could be updated.

A full-time faculty member, meanwhile, noted that the design of online courses requires substantially more time than the design of a physical course due to the technological requirements, and that the department should provide more support for design of online courses. Additionally, faculty expressed the wish that the number of times an online course design is utilized should be taken into consideration as a valuable faculty contribution when administrators conduct annual performance reviews.

The major recurring theme from the faculty interviews regarding technology was disillusionment with the university’s course management system, a software product that has been developed as part of an open-source code movement in partnership with several other universities. Many of the faculty bemoaned the loss of the course management system that was previously used, which was a product offered by a for-profit vendor. Two faculty members felt that the available technology was not adequate for providing effective courses, and were reluctant to teach courses online until the situation improved. It should be noted that the period of data collection was the first semester that faculty were required to use the new course management system, and therefore it is likely that some of the technology issues identified by faculty would be resolved as this change management initiative was further developed.

Twenty-three students responded to the online survey used for evaluation of the DM program. Among them, three were from the year 2000 group, four were from the 2001 group, and 15 were from the 2003 group. One student did not indicate the starting year group. (The 2002-2003 academic year was a transition period between the cohort and noncohort models, and no new students were admitted to the program during that time.) Other demographic information about the students who responded to the survey is summarized in Table 1.

Students who responded to the online survey identified two major factors for why they chose to study at the Midwestern Research University (MRU) DM program. These reasons were the overall reputation of quality of the department and full-time faculty, and the convenience and flexibility of the distance option.

Some respondents reported that they had researched other programs prior to choosing the MRU DM program, and had found the department ranking the best in the country. This indicates that, at least in its early years, the distance program was drawing students based in part on the quality associated with the residential program. While this is positive, eventually the distance program will need to establish and maintain a quality reputation of its own. As one student replied, the DM program should have courses taught by the world class faculty that make the residential program so well regarded. While some of these faculty members do indeed teach in the online program, others do not.

The student respondents also indicated that they chose the DM program because they could not move to a campus to complete the residential program, usually because of work and family commitments. While the distance option allowed for overall geographical and scheduling flexibility, once students were enrolled in the program, some discovered that the program was more traditionally structured than they had imagined. Some students found the time and dedication needed to complete the program more demanding than they had expected, and felt that the amount of group work was especially inflexible.

Students were asked to evaluate several aspects of the program related to technology use. Student responses were measured on a Likert scale with a low score of 1 representing strongly disagree and a high score of 5 representing strongly agree.

First, students were asked to describe their comfort level related to using technology before entering the program. Overall, students showed a high level of comfort with technology, indicating a high degree of readiness to engage in an online degree program (M = 4.22 SD = .850).

Overall, student ratings regarding the availability of technical support were relatively average (M = 3.70 SD = 1.105). Those few students who were low in technological readiness do appear to have received sufficient support where possible, as these students reported that peers and instructors provided learning resources, as well as assisting them directly. Those students who already were high in technological readiness perhaps did not need technical assistance unless the technical problem was systemic, in which case, the program technical assistants could offer little help (e.g., a university server was down).

Technology issues that were beyond the control of students seem to have affected them the most. Students rated the use of technology to support learning relatively low overall (M = 3.13 SD = 1.058). The most often cited technology issues involved both of the course management tools in use, the for-profit vendor system and the university’s open source system. The for-profit vendor system received mixed reviews, while the university’s open source system received almost universally negative reviews. One student bluntly commented, “[the vendor system] was weak but serviceable but [the university system] frankly sucks.” Given that the distance courses rely heavily on these tools for discussion, this is a significant finding. It may also be related to low community ratings for the program identified in other areas of the survey.

Other technology issues mentioned by students were the streaming video of residential guest lectures, PowerPoint presentations, and the library e-Reserves functionality. In each case, students appreciated the tools, but found problems in their implementation. For example, the quality of the streaming video was poor, and the audio narration accompanying PowerPoint sometimes did not work; without an included text narration, the bulleted slides did not support learning.

Students were asked to evaluate several areas of the program related to overall program management. Responses are shown in Table 2. Student responses were measured on a Likert scale with a low score of 1 representing strongly disagree and a high score of 5 representing strongly agree.

Overall, class registration in the DM program was the highest rated program management factor for each starting class. This indicates that registration did not seem to be an issue for most students in the program.

It is worth noting that registration was rated the highest factor for the DM program’s first class in 2000, when many of the university systems had not yet been configured for distance education, and the few university distance education courses that were offered were registered for differently than the department’s residential programs. This was partly due to program assistants occasionally developing “workarounds” within the university’s registration system to ensure that students were properly registered. As one student commented, “Registration is a confusing process though I’ve finally figured it out thanks to help from [the program assistant].”

Since the first cohort of the DM program enrolled in 2000, the university has undergone a comprehensive overhaul of its registration systems and software that has affected both residential and distance students. The DM program administrators and assistants are closely monitoring and accommodating online student registration needs during this transition.

Items related to general program support were the next highest rated components among the program management factors. Students reported that administrative staff, graduate assistants, and faculty were always responsive, especially in keeping students on task and offering feedback.

Students were asked to rate resources available in the DM courses. Overall, resources received average ratings (M = 3.39 SD 1.076). Many students found that the quality and quantity of the resources were valuable and appropriate. Obtaining the materials that were posted using the e-Reserves system, however, was frustrating at best. Students found that technical and administrative issues often prevented them from having the correct readings in time for a particular week’s assignment. Students often tolerated technical issues, but in the case of e-Reserves, they commented that since the course syllabus was prepared so far in advance, the system used for the course resources and readings should likewise be prepared and tested in advance.

Overall, the quality and amount of resources is acceptable to students; however, the overwhelming emphasis on the negatives of the e-Reserve system warrants special attention. While faculty and administrators laud the system for its assistance in copyright and fair use issues, it apparently has many technical limitations. Despite recent and ongoing improvements to e-Reserves, the negative comments on its technical issues were distributed over all three starting classes, not just the 2000 and 2001 classes.

Students were asked two questions related to academic advising. First, they were asked whether or not academic advising was important to them. This question was asked to consider the possibility that students in a distance program might be more selfdirected and therefore possibly desire less advising. Students rated their desire for advising at 3.83/5, indicating that advising was important to these students. Students were then asked whether they felt they received enough academic advising in the IST DM program. Overall, students are somewhat satisfied with the amount of advising (M = 3.22 SD .988). A couple of students commented that it would have been nice to have more interaction with advisors, using chat or some other medium.

The advising structure for the DM program has been changed since the program started. Initially, DM students were assigned individual faculty advisors, in the same way that residential students are assigned such faculty advisors. The 2003 starting class and all future starting classes, however, have the same advisor—the program director. Perhaps related to this change in advising structure, the 2003 starting class had the lowest rating for amount of advising (M = 2.93 SD = 1.486). As the number of students in the program grows, using only one person may be too overwhelming to provide adequate advising for students.

The orientation process has undergone major changes during the life of the DM program. The 2000 and 2001 cohorts participated in a mandatory 3-4 day residential orientation to the program. In 2003, the program changed its cohort structure, switching to a structure that allowed students to join the program at the beginning of any term rather than only at the beginning of the academic year. This change effectively ended the residential orientation.

The overall rating from all three classes for orientation was 3.04/5 (SD = 1.065). The 2003 class, which did not receive a residential orientation, rated their experience only as 2.53/5 (SD = 0.743). This was the lowest rated factor for this class in the program management area.

Given the changes regarding orientation, student reaction to the sufficiency of any program orientation might be understandably mixed. Students who had participated in the first cohort’s orientation enjoyed it. Students who were part of the second cohort complained about the timing of the orientation that was several weeks before the semester started. Finally, students in the 2003 group, who did not participate in any residential orientation, thought it would be beneficial. One student commented,

The need for a sense of community in online education has been cited often in the literature. This need is perceived to stem from the physical separation of learners from each other, the course instructor, and the program support staff.

The presence of this factor, however, was rated the lowest by students overall among any survey item in all survey categories. Online students did not feel they were part of a community within their online program, as well as within the department. In many cases, the DM students regularly received e-mails regarding residential activities that they, of course, could not attend. Students suggested that the program look for ways to integrate the DM students into the larger IST community, including a general discussion board for IST DM students, as well as a midprogram residential requirement, perhaps between the first and second years.

Students were asked to evaluate several areas of the program related to overall course management. The responses were shown in Table 3. Student responses were measured on a Likert scale with a low score of 1 representing strongly disagree and a high score of 5 representing strongly agree.

Students rated highly several online instructional design factors. These factors include the use of active learning strategies, an emphasis on student and faculty interaction, and the presence of clear learning objectives.

Most students found the courses well designed, and enjoyed the interaction with faculty and peers from around the country. Student comments support the quantitative findings that students are mostly pleased with the overall design of the DM courses At the same time, some students felt the amount of group work was inappropriate for a distance education program, and limited the program’s expected flexibility. The contrary comments indicate a tension between on one hand, what is acknowledged as “good” instructional design for distance education courses, and on the other hand, the flexibility needed by working adults in academic programs with above average quality expectations for graduate study. This tension was also noted in an interview with at least one online faculty who noted that, all other academic factors being equal, achieving the highest quality in graduate studies involves devoting significant amounts of time—which working adults may not be able to contribute.

Adequate and timely feedback from instructors is important in any educational setting. In an online environment, given the physical separation of learners from the course instructor, the need for feedback is even more apparent.

Students were asked to rate the timeliness and the adequacy of instructor feedback in the DM program. Both factors received the same overall rating of 3.78 (timeliness SD = 1.126; adequacy SD = 1.043). Most qualitative comments focused exclusively on the timeliness issue. The students who commented indicated that they were often working on the next phase of a project before they received instructor feedback on a previous phase. In some cases, instructor feedback occurred more than 2 weeks after an assignment submission. Peer feedback, while valuable, was not perceived by students to be as beneficial as instructor feedback.

Distance education has held the promise of providing individualized evaluation that can allow students to show their understanding of material in myriad ways different from traditional methods. Students, however, rated this as the lowest factor among all course management factors (M = 3.39 SD = 1.033). One student commented, “I expected more variety in the methods of collaboration and evaluation. Writing alone, whether forum postings or papers, seems somewhat limited for a program devoted to instructional systems. There are a lot of other options out there.”

Students were asked to evaluate several areas of the program related to overall quality. Overall, students rate favorably the quality of the instructional design and learning effectiveness of the DM courses. This aspect of the program rates higher in general than the several program management factors and the program technology aspects. One student commented “Overall, I am very happy that I chose the DM program ... the program was an exhausting three-year ride for me, with some disappointments, but I certainly have no regrets.”

The evaluation results of the DM program discussed earlier are summarized in Table 4. In all, students rated favorably the quality of the DM program. They were satisfied with the class registration process, the overall instructional design factors of the online DM courses, the quality and quantity of the resources they were provided, the feedback from the faculty, and the support they received from the administrators and faculty. In the meantime, they hoped some areas could be further improved, especially regarding advising, timeliness of feedback, variety of feedback, and sense of community.

After looking at the data collected from the administrators, faculty, and students as a whole, two factors among others are especially worth discussing. Corresponding recommendations are given below based on relevant literature and issues discovered during the course of this evaluation.

As well documented in the literature, technology plays a critical or fundamental role for successful Internet-based distance programs (see for example, Bonk, 2001; Zhai & Liu, 2005). Results of this evaluation showed that technology is a powerful concern for the major stakeholders of the DM program. The administrators reported that technology hindered the program in many aspects such as registration, administration system, and course management. The faculty noted that the course management technologies were not adequate for providing effective online courses. While students of the DM program showed high technology readiness, they were not very satisfied with the uses of technology to support their learning.

The current course management system provided by the university was a frequent target of criticism. Faculty expressed reticence at teaching with the university’s open source system, and one student represented the feelings of many others when he frankly commented that the current management system “sucks.” Considering the importance of the course management system, it would be beneficial for the DM program to work more closely with university technology services to improve the tools.

Overall, students were relatively satisfied with the technical support provided by program assistants, though this might be in part because of students’ high technology readiness. Adjunct faculty showed overall satisfaction with support from graduate assistants, though some faculty would like to have more direct control of the courses. Program administrators need to take a more proactive role in getting support from relevant parties of the university in the areas that are beyond their direct control, such as the e-Reserve system provided by the library and the course management system provided by the university.

Faculty members play a key role in using technology successfully in online courses, and their participation is believed to be inseparable for successful online programs (see for example, Garrison, Anderson, & Archer, 2003; Schifter, 2004; Liu, 2005).

All three groups identified the reputation of the department and its faculty as the most attractive feature of the program. Student respondents indicated that the reputation of the full-time faculty was a major reason why they were attracted to the DM program. While the quality of the adjunct faculty was deemed to be high by administrators and the full-time faculty, it would seem that students who are coming to the program with the expectation of being taught by nationally well-known faculty members are not having their needs met. While the program focuses on having a full-time faculty member teach each year, it seems important to maintain or even increase the involvement of the full-time faculty as this impacts the program’s attractiveness and directly addresses student needs.

The DM program has employed more adjunct faculty members to teach the courses since 2002. Consequently, the adjuncts play more important roles in the success of the DM program. The issues related to the adjunct faculty need to be given more attention by administrators. The adjuncts do not have much control over the courses that they teach. The courses designed for the distance program are based on the residential versions, and designed by the full-time faculty. While full-time faculty might redesign courses that they teach and review the program yearly to ensure that all courses are covering what they should be, the adjunct faculty largely felt that they did not have the freedom to do anything other than make extremely small adjustments to the courses that they were teaching. Some of the adjuncts had issues with current course designs, but the point was made several times that there is little incentive for the adjuncts to redesign the courses. Additionally, the adjuncts noted that they often have to go through several different cumbersome processes to make any changes to the courses, such as for technological problems, an adjustment to the syllabus, or the materials set aside in the electronic reserves system. Adjunct faculty should be empowered with more direct control over the courses they teach.

Economic considerations appear to have some impact on the program and its effectiveness. The use of adjunct faculty, as well as the cessation of the cohort program and the distance program’s orientation, all appear to be influenced by the economics of the program. Additionally, course design appears to be affected, as there is little incentive for either adjunct or full-time faculty to redesign courses. These choices surely have an impact on the quality of the program, which is evident from the fact that program management as a whole, including orientation, support, advising, and sense of community, was not highly rated by students. For this reason, it seems that it would be a very valuable exercise for the DM program administrators to gain a comprehensive understanding of the economics, including both the costs and benefits, of maintaining the distance master’s program.