Using Important-Performance Analysis To Guide Instructional Design Decisions For E-Service Learning

Service-learning has been a proven learning strategy from the days of Dewy and is often intertwined with other forms of experiential learning, such as practicums and work-related learning. Caspersz and Olaru (2015) define service-learning as a “process of reflective education in which students learn civic or social responsibility through a scholarship of community engagement that embodies the principle of reciprocity” (p. 19). In practice, service-learning integrates academic rigor with real-world projects that students undertake with community groups and thus creates a reciprocal learning relationship between the students and their community groups. Typically, community-engaged pedagogies such as service-learning require facilitative expertise that educators develop and refine through scholarly teaching that integrates experience and reflective teaching (Strait et al., 2015; Waldner, 2015).

Service-learning is often integrated into face-to-face environments, but creating highimpact practices for online service-learning is imperative with the number of online courses and programs. E-service learning occurs when either the instruction, service, or both components are conducted online. For example, students in an online history class may assist a museum with digital archives. Dailey-Herbt and colleagues (2008) described e-service learning as a pedagogy that engages learners through technology. Previously, it was not easy to engage the community from a distance, but technology development, connecting students from a distance with community-based practices, is seamless.

With the increase in online education, many studies focus on transitioning from traditional service-learning to e-service learning. Often educators find the online environment as a barrier to effectively implementing service-learning. Frameworks have been proposed for implementing e-service learning, yet there is no 𠇉one size fits all” due to the diverse needs of service projects. Researchers hope that the design and development of e-service learning will help future educators make informed decisions and plan the best implementation for their respective projects. E-service learning challenges educators to make full use of technology rather than replacing instructors with technology. The focus should be on the design of the course and project.

Although there is a plethora of research regarding service-learning benefits, research on online service-learning is lacking. Marcus and colleagues (2020) conducted a systematic review of e-service learning literature and found only eleven studies that focused on the design and development of service-learning for online courses. Many of the studies offer suggestions for designing e-service learning for various disciplines. Strait and Sauer (2004) provided suggestions for beginners, such as starting small with one course, introducing learners to the purpose, providing explicit communication channels for both the learners and community partners and ensuring flexibility. Bangert (2004) focused on the reflective portion of service-learning, stating that discussions should be carefully crafted to create stimulating discourse and groups monitored to ensure equal participation. Helms and colleagues (2015) also focused on group communication. They recommended creating a group charter for students to develop a plan for working together, including developing a communication plan. Students may have to identify community partners or clients since many students may live outside the school area they are attending; it was also suggested to provide a list of criteria the project must meet. Students should submit a proposal early in the semester with the organization and explain why this organization meets the criteria.

Overall, the best practices suggested by researchers are also best practices for online course design and development, as suggested by Quality Matters (QM). QM. is a faculty-oriented, process-centered peer review process developed from research based on instructional design to ensure quality design in online and blended courses. Many of the suggestions for designing e-service learning courses are incorporated within the QM rubric’s best practices. For example, introducing the learners to the purpose is covered in specific standard 1.2. Creating engaging and meaningful discussions is covered in specific standard 5.1. Specific standard 5.4 addresses communication among learners. Although researchers have offered some specific examples, such as using a group charter (Helms et al., 2015), designing a course to meet QM standards can significantly assist an instructor in any discipline with designing and developing a course to support e-service learning.

This study analyzed student perceptions of an e-service learning project to inform future course design.

This quasi-experimental study explored students’ perceptions regarding online experiential learning in terms of what they perceived to be essential and how they performed. This study utilized an importance-performance analysis (IPA) method. Additionally, students were required to complete a reflection at the end of the semester. The reflections were analyzed to garner additional information about the students’ experience with service learning in an online asynchronous class.

This study used a graduate course in an instructional technology program delivered in spring 2020 during a 15-week semester. The online asynchronous course had 14 students. Students were recruited to participate in this study voluntarily at the beginning of the semester. Students who chose to participate in the study were assigned a random identification number, which allowed the researcher to match survey data anonymously. Twelve students completed the importance and performance survey, with an 85% response rate. This response rate falls in line with the recommended response rate for online surveys (Baruch & Holtom, 2008).

Importance-performance analysis has been used primarily in marketing and tourism yet has found a place in higher education for student evaluation of teaching and course design (Anderson et al., 2016; Caspersz & Olaru, 2015; Huybers, 2014). Alberty and Mihalik (1989) also stated that IPA has an important place in adult education. Important-Performance surveys allow students to evaluate both constructs of various teaching attributes (see Figure 1). Specifically, the importance-performance model identifies the relative importance of factors influencing student preference (importance component) and indicates how a particular instructor or setting possesses these factors (performance component).

The importance survey, designed to implement at the beginning of the semester, could also be considered a presurvey. The performance survey deployed at the end of the semester is considered a postsurvey. For this paper, the surveys will describe importance and performance (pre/post).

The SELEB (Service-Learning Benefits) survey developed by Toncar and colleagues (2003) was used to gauge student perceptions of service-learning with minor word changes to terminology (Caspersz & Olaru, 2017). The SELEB scale consists of 15 items that fall under four main factors: practical skills, interpersonal skills, citizenship, and personal responsibility. The Importance SELEB scale was given to the students at the beginning of the semester. The scale is 1-5, with 1 being extremely important and 5 not important. The Performance SELEB was deployed at the end of the semester upon completing the experiential learning activity. Table 1 displays the survey items and how each is associated with each main factor. There are four factors associated with the 15 variables.

The project was implemented in a fully online asynchronous graduate instructional technology course on change management. The course was designed to align with the QM rubric and was informally reviewed by a certified QM Peer Reviewer. Students were asked to reflect on their experience every 3 weeks with a cumulative reflection at the end of the course. The students were also expected to complete a service-learning project for a local nonprofit organization.

The class teamed up with a local nonprofit to assist with implementing a technology change within the organization. The instructor chose the community partner, and students formed teams to tackle the problem collaboratively. Students were required to attend a kickoff meeting (synchronously) with the administrative staff from the nonprofit organization to review the organizational needs. Since the entire class was working with one client, it was determined that all communication, including questions, would go through the instructor, and questions would be sent to the client once a week on Thursdays with an expected response time on the following Tuesday. The students were required to submit various checkpoints for their final change management plan throughout the semester.

IPA was used to analyze student perceptions and Performance. This data was used to construct a two-dimensional matrix. Importance is

depicted on the y-axis, and Performance is depicted on the x-axis. The IPA graphs' gridlines are determined by the overall mean of Importance (4.40) and Performance (4.23).

A paired-samples t test was conducted to evaluate the impact of the experiential learning assignment using the importance-performance scale. There was no significant difference between importance and Performance for all associated variables. Table 2 displays the mean scores for the four main factors of the SELEB survey. There was a decrease in the mean from importance to performance in interpersonal skills and citizenship and increased personal responsibility.

A paired-samples t test was conducted to analyze the four main factors comparing the importance to the performance. Practical skills remained the same from importance (M = 4.58, SD = .452) to performance (M = 4.58, SD = .452), t (9) = .000, p = .1.000 (twotailed). The mean decrease in the practical skills scores was -1.293 with a 95% confidence interval ranging from .458 to 2.112. The eta squared statistic (0) indicated a small effect size. Interpersonal skills resulted in a decrease from importance (M = 4.52, SD = .444) to performance (M = 4.33, SD = .577), t (9) = 1.644, p = .139 (two-tailed). The eta squared statistic (.37) indicated a moderate effect size. Citizenship skills resulted in a minor decrease from importance (M = 3.83, SD = .750) to performance (M = 3.80, SD = .933), t (9) = .096, p = .926 (two-tailed). The eta squared statistic (.035) indicated a small effect size. Personal responsibility skills resulted in an increase from importance (M = 3.89, SD = .943) to performance (M = 4.22, SD = .687), t (9) = -1.664, p = .135 (twotailed). The eta squared statistic (.40) indicated a small effect size (see Table 3).

Table 4 depicts the analysis of the twotailed t test for the variables of the four main factors. Each item is labeled, depicted in Figure 2, on the IPA quadrant. There is a decrease in the mean scores from importance to performance.

Figure 2 illustrates the associated variables from the importance and performance SELEB scale on an importance-performance quadrant. The x and y-axis are determined by determining the overall mean score for both importance and performance (Huybers, 2014). The overall importance mean was 4.40 and overall performance was 4.23. Applying real-world skills (A), practical workplace skills (B), organizational skills (C), personal growth (F), self-confidence (K), and problem-solving techniques (N) landed in the upper right quadrant, which suggests that this activity met both the importance and performance for students. Leadership (L) and development of oral and written skills (M) fell in the lower right (concentrate here) quadrant. Greater understanding of culture and racial differences (D), social responsibility (E), community involvement (G), make a difference in the community (H), view social issues from a variety of perspectives (J), and demonstrate trustworthiness to trs (O) fell in the lower left (low priority) quadrant.

There are many reasons for implementing service-learning to enhance student learning outcomes and various tr skills, such as the development of social responsibility. Within the online environment, service-learning can prove challenging. Incorporating IPA can assist the course designer/instructor with design suggestions focusing on specific skill development. For example, leadership and oral and written communication skills were an area that needed concentration. The course designer can take this information to enhance student learning in these areas. Each student was in a team for this course; therefore, many of the students should indirectly be working on both leadership skills within the group and the client and their oral and written communication. The course designer will revisit these areas to develop more concrete examples of the expectations of working with a team of peers and how to interact with a client to ensure the students stretch their knowledge and improve already existing skills. With the areas that fell into the low priority quadrant, the course designer/instructor needs to analyze the skills and determine if they are pertinent to the course/program goals. An instructor can focus on specific areas for student development through IPA and build those specific concepts into the course design.

Service-learning allows students to gain additional skills beyond the classroom concepts. In this study and trs using IPA for an e-service learning project, the students indicated strong application of classroom concepts to a real-world application (Anderson et al., 2016). At the very least, students indicated high satisfaction in this area.

There were a few limitations with this initial study. First, it was a small class, but as Strait and Sauer (2004) stated, it is important to start small and then apply appropriate design to new courses. Next, during the middle of the semester, the COVID-19 pandemic put additional stress on many students either due to adding homeschooling to their daily activities and graduate school work and full-time employment. During the last half of the semester, many students focused their reflections on venting about COVID-19 and the undue stress that came with it. Some of the COVID-19 stress/fatigue affected group dynamics. Finally, the class was given a project to work on rather than choose their own project, which could have influenced the degree the students felt invested in the project and with the client.