The GreenComp Evaluation Roadmap: a roadmap for in-depth holistic evaluation of the integration of the GreenComp framework in higher-education curricula

To tackle the complexity of the 21st century’s sustainability challenges, curricula should transition beyond the sole transfer of topical knowledge and discipline-specific skills, toward the transfer of such knowledge and skills integrated with the development of key sustainability competences. This will increase students’ capacity to take ownership of the sustainability agenda and empower them to become agents of change (Boeve-de Pauw et al., 2015; UNESCO, 2017) within their local context (Lozano et al., 2017).

The concepts of education for sustainable development (ESD) and sustainability education (SE) capture the need to teach learners such key sustainability competences, aiming to generate change agents, problem solvers and transition managers (Wiek et al., 2011), individuals essential in guiding us toward achieving the sustainable development goals and beyond.

Varying visions and definitions for “sustainability” and “competence” in the literature revealed a need develop a “widely accepted competence framework” (Bianchi, 2020, p. 5). Relying largely on four major works in this discipline (Brundiers et al., 2021; Redman and Wiek, 2021; Wiek et al., 2011, 2016), in combination with insights from 15 other ESD and ES frameworks, and extensive stakeholder consultation, Bianchi (2020) created a synthesis identifying key competences, knowledge, skills and attitudes, forming the basis of the GreenComp framework (Bianchi et al., 2022). This overarching framework, aims to “provide a common ground to learners and guidance to educators, providing a consensual definition of what sustainability as a competence entails” (Bianchi et al., 2022, p. 2). According to the framework, sustainability is a competence of itself, consisting of 12 subelements referred to as “competences for sustainability” (Table 1). These 12 competences for sustainability of the GreenComp framework are grouped into four areas. Each competence is further subdivided into a number of knowledge, skills and attitudes (KSA) statements (Bianchi et al., 2022). While other competences for sustainability have been identified (e.g. Brundiers et al., 2021, 2019; Redman and Wiek, 2021; UNESCO, 2017; Wiek et al., 2011), in the remainder of this article the term “competences for sustainability” will refer to these 12 competences as the “competences for sustainability” in the further text.

While it is indicated that framework can be used for self-assessment, reflection and the review of a curriculum, the development and description of a step-by-step method to conduct such self-assessment of and reflection on a curriculum was beyond the scope of the previous study (Bianchi et al., 2022). Researchers have previously reported on other tools and methods to assess the integration of sustainability competences in curricula or programs, including the Campus Sustainability Assessment Framework (Cole, 2003); Auditing Instrument for Sustainability in Higher Education (AISHE; Roorda, 2002), which has now been further developed into Sustainability Higher Education (SHE; Leerdam and van Gulik, 2020); Graphical Assessment of Sustainability in Universities (GASU; Lozano, 2006); Sustainability Tool for Assessing University’s Curricula Holistically (STAUNCH; Lozano and Peattie, 2011; Lozano and Watson, 2013), a self-evaluation tool developed by Wyness and Sterling (2015), or more complex tools such as the uncertainty-based quantitative assessment of sustainability for higher education institutions (Waheed et al., 2011). Most of these tools use indicators, which are scored by either lecturers, program managers, students and/or external consultants, or the tools use questionnaires to be filled out by lecturers and/or students. It is often reported, however, that in-depth insights from these tools remain limited by the lack of understanding of the true meaning behind the scores or answers to the questionnaire. For example, it was acknowledged that the audit of the STAUNCH tool should be complemented with staff interviews, as SD education practices which are not incorporated in the course documentation, are taken into account (Glover et al., 2011; Watson et al., 2013). In the AISHE method, this shortcoming is partly addressed via a consensus discussion at the end of the evaluation (Lambrechts and De Prins, 2008). Also Ceulemans et al. (2011) included interviews within their assessment to provide more in-depth information (Ceulemans et al., 2011).

While the GreenComp framework has been applied in previous studies (e.g., Latham et al., 2023; Piciga, 2023; Sourgiadaki and Karkalakos, 2023), this has not yet yielded a step-by-step approach to use the framework for the evaluation of an entire curriculum. A tool developed by Dumitrache et al. (2022) evaluates professional training in Romania based on the GreenComp framework, but the evaluation is solely based on the course descriptions. This limits insights into how the courses are taught in practice, such as examples used in class, and assignments given to the students to support their learning (Lozano and Peattie, 2011; Watson et al., 2013). Insight into such day-to-day practices gives a more in-depth representation of how the competences for sustainability are integrated in a curriculum and allows for the identification of potential gaps (Annelin and Boström, 2022; Ceulemans et al., 2011; Glover et al., 2011; Makrakis and Kostoulas-Makrakis, 2016; Wyness and Sterling, 2015).

In this study, this gap in the literature is addressed by building on previous approaches to curricular assessments. Specifically, the GreenComp Evaluation Roadmap (GER) is presented, which aims to evaluate the extent to which the competences for sustainability and KSA statements of the GreenComp framework are integrated in a curriculum. The roadmap consists of a mixed-method approach, with simultaneous hybrid data collection, allowing for a thick description of a curriculum, including its content and its daily teaching practices, and to explore more complex aspects that do not arise from quantitative indicator scoring or questionnaire responses (Leydens et al., 2004; Makrakis and Kostoulas-Makrakis, 2016; Malina et al., 2011). Applying the GER to an existing case study, this article also aims to detail if and how the roadmap can be an addition to the already existing sustainability assessment tools.

The GER comprises three steps: Step 1: simultaneous hybrid data collection through semistructured interviews guided by a radar chart-based rating of the learner competences for sustainability for each course; Step 2a: a qualitative analysis of the semistructured interviews; Step 2b: statistical analysis of the data resulting from the rating exercise; and Step 3: focus group to validate the previous steps (Figure 1).

Step 1 entails a simultaneous hybrid data collection through semistructured interviews guided by a radar chart-based rating of the learner competences, which allows for the collection of quantitative data (through scoring) and qualitative data qualitative (through semistructured interviews).

While, semistructured interviews generally follow an interview guide (Bernard, 2006), here data collection is guided by a radar chart and a scoring board (Figure 2 and Annex 1). Before the data collection session starts, respondents are informed about the goal of the research and about the GreenComp framework. It is important to emphasize that the data collection is focused on their perception of the alignment of the course with the GreenComp framework, but is not an evaluation of the course quality, nor of the lecturers themselves.

The radar chart is a four-level dodecagon with the 12 competences for sustainability on the corners. After reading the competence descriptor (Table 1), the respondents are asked to rate how well the competence is integrated in their course on a scale from 0 to 4 on the radar chart (0 = not taken into account, 1 = slightly taken into account, 3 = taken into account, 4 = taken well into account). Given the semistructured nature of the data collection method, the researcher can ask the respondents to further explain their scores, and provide examples of their teaching practices.

In a second phase of Step 1, the respondents are asked to select the one competence that is most prominently incorporated in their course. For this competence, they are presented with a stack of cards containing its associated KSA statements and asked to rank these cards from “not integrated” to “highly integrated”. As with the radar chart exercise, the respondents are asked to explain their choices and provide examples.

All data collection sessions are recorded so they can be transcribed and analyzed (Step 2b, Section 2.4) and at the end of each session a picture of the charts is taken for further analysis (Step 2a, Section 2.3). A data collection session should be conducted for each course of the curriculum under evaluation, and takes about 0.5–1 h per course discussed.

In Step 2a, the scores provided by the respondents in Step 1 are digitized for statistical analysis, which focuses on analyzing variation within the curriculum, using individual courses as units of observation. Hence, the study population would be all courses within the curriculum. It is recommended to include all the curriculum courses in the study (i.e. the entire population) rather than taking a sample of the courses. If time and resources only allow for a sample of courses to be considered, it would be recommended to stratify across curriculum years and course types (mandatory courses, specialization courses, electives), to ensure accurate representation of curriculum components. The analyses can then be organized into a descriptive and an exploratory component:

• Descriptive statistics:

  • Median overall scores per year, specialization or alternative subdivisions within the curriculum.

  • Average overall scores per year, specialization or alternative subdivisions within the curriculum and average GreenComp area scores per year, specialization or alternative subdivisions within the curriculum.

• Exploratory statistics:

  • Cluster analysis of competence scores to identify courses that foster competences for sustainability in similar ways.

  • Principal component analysis (PCA) to extract summary variables, (linear) combinations of competences for sustainability scores, allowing to quantitatively show similarities and differences between courses, or groups of courses.

  • Average score of KSA statements per course to identify whether there are any imbalances in the curriculum regarding its focus on developing knowledge, skills and/or attitudes.

Annex 2 contains the template spreadsheets and markdown file for the quantitative analysis.

To analyze the qualitative data from Step 1, the recordings of the data collection sessions should be transcribed, coded and analyzed, using software such as NVIVO. Coding can be done by competence, with subcoding for year and/or specialization track and course. Cases can be made per lecturer to be able to identify any response bias or “teacher effect” in responses, whereby some respondents are naturally inclined to give higher scores than others. Annex 3 presents an example codebook for the analysis.

Step 3 is to organize a focus group session preferably with all lecturers/respondents. The focus group session preferably takes place a few months after the simultaneous hybrid data collection round (Step 1) aiming to validate the results of Step 2a and Step 2b. Therefore, during the session, the attendees are asked for feedback on the results presented. In addition, they are consulted on whether they have redesigned their course in the period between the data collection sessions and the focus group, or are planning to do so in the near future.

It is recommended to limit the duration of the focus group to 2 h maximum, as the attention drops significantly after this time span (Nyumba et al., 2018). Annex 4 presents a draft which can be adjusted according to the local context.

The GER was applied to the Sustainable Island Solutions through Science, Technology, Engineering and Mathematics (SISSTEM) Bachelor Program taught at the University of Aruba (UA). Before results are discussed (Section 3.2), some contextual background is provided (Section 3.1).

SISSTEM – Sustainable Island Solutions through Science, Technology, Engineering and Mathematics (STEM) – is an academic education and research program at the University of Aruba aiming to educate local and regional youth in STEM at tertiary level, focusing on finding solutions for sustainability issues in Aruba and other Small Island States (SIS) ^[1]. Faced with a complex and intertwined set of economic, social and environmental challenges, Aruba needs a systemic shift, including a shift in its economy, education and workforce with a more holistic skillset to address the island’s sustainability challenges (Hermans and Kösters, 2019). In this context, SE is crucial, and has already proven to have significant impact on the students at the University of Aruba (Eppinga et al., 2020).

In our case study, the authors focus on the SISSTEM Bachelor Program, which was developed with the SE concept in mind, and whereby its developers aimed to have a balanced mix between fundamental STEM courses, and SE courses. Figure 3 shows an overview of the SISSTEM bachelor curriculum. For an in-depth discussion of the SISSTEM program, the authors refer to Mertens et al. (2023).

Using the GER, the authors evaluate the integration of the 12 competences for sustainability in the curriculum.

Following Step 1 of the GreenComp Evaluation Roadmap, 14 data collection sessions ^[2] were conducted. In total, the lecturers scored 27 out of the 29 SISSTEM courses, as well as the international exchange semester, representing 94% of the curriculum credits.

Generally, the authors observe progressively more attention for the 12 competences for sustainability as the curriculum proceeds (Figure 4). Once the topical knowledge and skills have been established in the first part of the program, the curriculum leaves increasingly more room for the development of competences for sustainability. This insight is also confirmed by the lecturers. For example, one lecturer stated: “So once we establish the foundation in chemistry, we can focus a little bit more on sustainability in environmental chemistry.”

The exception to this general trend is the competence “Supporting fairness” showing a slightly higher median score in Ba1 (0.5) compared to Ba2 (0) (Figure 4). This can mainly be attributed to the courses of the first semester of Ba2, which have a median score of 0 on this competence. The median score of all specialization tracks for this competence is 0.5. Another exception to the progressive growth in competences for sustainability scoring is the specialization track “Technology and engineering” (Figure 5). Here, it seemed that the lecturers initially focused more on topical knowledge and design and implementation of sustainability technologies and textbook examples and exercises, rather than integrating competences for sustainability and contextualization.

The general trend of increasing attention for the competences for sustainability was also explicitly recognized by the lecturer of Int. Appr. stating:

There is a strong focus on the competences for sustainability in the area “embracing complexity” throughout the entire curriculum. The area “embodying sustainability values” scores relatively low overall, except in the specialization track bio-environmental sciences and Ba3. Furthermore, the areas “envisioning sustainable futures” and “acting for sustainability” score relatively low, except in the specialization track information and data sciences and Ba3.

Figure 6 presents the results of the cluster analysis, which shows four clusters. One cluster, containing four courses, is characterized by relatively high scores on all competences for sustainability. These courses can be categorized as the SE courses of the curriculum (indicated in ochre in Figure 6). Similar to this former cluster, another cluster of four courses is characterized by lower overall scores on the different competences for sustainability. Courses in this cluster, however, still score relatively high on competences for sustainability 4–6 (area “embracing complexity”) and relatively high on competencies 1–3 (area “embodying sustainability values”), as well as competences for sustainability 9 (exploratory thinking), 10 (political agency) and 12 (individual initiative). These courses are categorized as the hybrid SE courses of the curriculum (indicated in burgundy in Figure 6). Another cluster of 5 courses shows lower average scores than the preceding cluster, with a disproportionate decrease in attention for the competences for sustainability 2 (supporting fairness), 4 (systems thinking) and 10, but relatively high attention for the competence 12. Even so, this cluster shows that the areas “embracing complexity” and “envisioning sustainable futures,” together with competency 12, receive substantial attention in the courses within. Finally, the fourth cluster is large, including 12 courses, which have still lower average scores as compared to the previous cluster, with the exception of competency 10, but a disproportional decrease in attention for competency 12. Given the relatively lower scores of the courses in this latter cluster, they are categorized as the foundational, topical knowledge and skills development courses.

The observation that the four clusters mainly differ in overall scores on the competences for sustainability is corroborated by a PCA (Figure 7). Here, the scores were standardized, so that each competence had equal weight in the analysis. Varimax rotation was used to ease interpretation of the first two PCA components. Specifically, the PCA identified two components, which together explained 80% of the variance in assigned competency scores. The loadings of all competences for sustainability were positive on both components, suggesting that courses situated in the upper right quadrant of the principal component plot were characterized by higher scores on all competencies. By averaging the loadings of courses for the three bachelor curriculum years, a clear progression was found toward courses with more attention for the sustainability courses, the biggest change occurring between Year 2 and Year 3. Here, it should be noted, however, that the number of courses in Year 3 is small (n = 2). Nevertheless, the shift from foundational courses to applied sustainability courses was clearly reflected in the ordination of course competency scores (Figure 7).

Figure 8 shows the average KSA statements scores for different courses. There is no over- or underrepresentation of either knowledge, skills or attitudes, signifying a sound balance between them throughout the courses and the curriculum. The exception is “Chem. Sep.” which was scored for the competence “Critical thinking”. Here all knowledge statements scored 0, and the attitude and skills statements scored low. This course is one of the lowest scoring courses in the entire curriculum overall (average score of 1, score of 2 on critical thinking). Hence, it is not surprising that the course does not align well with the KSA statements either.

A focus group was organized about 10 months after the data collection sessions. There was a representative group of seven lecturers present at the focus group itself, and the results were discussed individually afterwards with two additional lecturers. During the focus group, the results presented in Section 3.2.1 were discussed via interactive questions (Annex 4).

During the discussions, about half of the respondents stated that after the data collection session they reflected a lot on the competences for sustainability in the context of their courses. In addition, almost all respondents stated that after the data collection session, they made changes to their course design (in assignments, examples, exercises) to align better with the competences for sustainability. Already during the data collection sessions, respondents hinted at this. One respondent stated: “But I could see how I could twist it now for next year.” One respondent indicated he did not make any changes, because of the already strong alignment of his course with the competences for sustainability.

Finally, when asked which of the more “underrepresented” competences for sustainability deserved more attention in the curriculum, “valuing sustainability” and “adaptability” were ranked as first and second, respectively, followed by “supporting fairness,” “collective action” and “political agency”. Several options were discussed as to how to better integrate these competences for sustainability in the curriculum. Overall, the respondents showed a positive attitude toward the framework and the evaluation exercise.

Besides allowing the identification of the well-represented and underrepresented competences for sustainability in SISSTEM, applying the GER also allowed to identify benefits and limitations of the approach.

First, during the data collection sessions it became clear that for some lecturers scoring their course(s) for the different competences for sustainability was not always straightforward. Generally, the lecturers could understand and align well with the descriptors of the “embracing complexity” competences for sustainability. However, some lecturers were not familiar with the vocabulary used in other descriptors, making them feel a bit alienated from those competences for sustainability. One lecturer, with a background in computer sciences, stated “For me, it’s hard to map it directly […] The language of the social scientist is different from the language of the engineer.” Recognizing this difference in language is therefore crucial during the data collection sessions. This challenge has also been recognized by Wyness and Sterling (2015), who stated that while they gave brief definitions of the concepts in their questionnaire, participants still stressed their lack of knowledge and therefore the subjective nature of their responses. However, because of the semistructured data collection set-up of the GER, participants could ask additional questions on the definitions provided and the researcher could ensure that the definition was interpreted correctly by the respondent.

Second, respondents sometimes felt uncomfortable when giving low scores, which can be attributed to the direct interaction with the researcher. This occurs less with more distant assessment tools like AISHE and STAUNCH. While it was stressed at the beginning of the data collection session that it was not an evaluation of them as lecturer, nor of their course as such, during the data collection sessions some respondents repeatedly apologized when they gave low scores. One respondent used the word “sorry” at least 12 times during a single data collection session for giving low scores. Respondents should feel sufficiently safe during the data collection session to score their courses based on their genuine judgment, rather than feeling compelled to give higher scores to avoid any sense of discomfort.

In addition, integrating the competences for sustainability in the curriculum will take more time for some courses, as lecturers need the right topical knowledge, and have sufficient background on Aruba and the SIS context, to really be able to teach the students the contextual setting of the knowledge acquired. Sufficient time is needed to develop such contextual knowledge. Additionally, lecturers should be acquainted with the principles of ESD or ES and the competences for sustainability. As one lecturer stated:

Having sufficient skilled teachers with experience of working with ESD has indeed been found to be one of the success factors of implementing ESD (Holmberg et al., 2008), but requires time to develop.

Third, lecturers risk, especially after giving low scores, to overcompensate and exaggerate in integrating the competences for sustainability. As unlimited course redesigns may inadvertently disrupt the alignment of the curriculum, a constant balance needs to be maintained between developing topical knowledge and skills and competences for sustainability. The tension between “too much engineering” and “too much social sciences” in an engineering curriculum is not unique for this program and was also found in other ESD initiatives (Bryce et al., 2004; Holmberg et al., 2008; Kjellgren and Richter, 2021; Mulder, 2017). The focus group could be used to detect such overcompensation.

Finally, response bias or “teacher effect” may exist, whereby some individuals are naturally inclined to give high scores, while others are naturally inclined to give lower scores. During reflections on the GER, it was suggested that it could be an improvement to allow the respondents to distribute a maximum cumulative score over the different competences for sustainability, or to correct to the same mean values. However, the main goal of the GER is to gain qualitative insights more than quantitative insights about the integration of the competences for sustainability in a curriculum. In addition, large deviations in scoring due to the potential teacher effect could be further explored and discussed during the focus group.

The GreenComp framework serves as common ground for learners and educators participating in higher education curricula to implement sustainability education (Bianchi et al., 2022). The framework is also intended to be used for self-assessment and reflection on and the review of curricula. However, as such an in-depth evaluation tool for the GreenComp framework was missing, a specific method was developed to evaluate the extent to which the 12 competences for sustainability and associated KSA statements are integrated in a specific curriculum.

Here, the authors aim to address this gap by presenting a simultaneous hybrid data collection through semistructured interviews guided by a radar chart-based rating of the learner competences for sustainability for each course. Often, the reason for not integrating qualitative data to generate more in-depth knowledge in tools such as STAUNCH is the time-investment needed. While the data collection sessions and the focus group of the GER require some time-investment, a lot of information could be retrieved from it. The value of the roadmap lies in its combined use of the strengths of both qualitative as well as quantitative results, leading to a more in-depth analysis (Ceulemans et al., 2011; Gobo, 2015; Makrakis and Kostoulas-Makrakis, 2016; Malina et al., 2011). In addition, by providing the template spreadsheets and markdown file for the quantitative analysis (Annex 2), the codebook for the qualitative analysis (Annex 3), the data analysis is being made more accessible and methodical for potential future users of the roadmap. Additionally, while it is not always easy for all lecturers to align with the GreenComp vocabulary, the personal approach through the semistructured interviews during the hybrid data collection sessions makes it possible for them to provide a score for each competence and explain their choice.

The quantitative results of the GER can serve as a baseline insight into the extent to which and how the different competences for sustainability are integrated within a tertiary curriculum. Repeating the roadmap exercises over time, could lead to an evaluation of the evolution of a curriculum toward incorporating the competences for sustainability over time. At the same time, the authors would not recommend using the GER as a comparative tool between different programs. While the scores provide insights, they cannot be interpreted as fixed values as they are a snapshot of the lecturers’ judgments and can also be highly context and time dependent. One should therefore be careful not to get tempted to overinterpret the quantitative results. Testing the roadmap on the SISSTEM bachelor program case study, the authors found that it guides an in-depth evaluation of the curriculum, allows for reflection and enables the identification of competence gaps. Hence, the GER can be a great addition to the already existing sustainability assessment tools, from which future users can chose in relation to their time, resources and intended depth requirements of the assessment.

While other authors (Kalsoom, 2024; Vare and Scott, 2007) have stressed that the research on ESD should also be shifting its focus from promoting skills and competences for sustainability in learners, toward really changing the learners toward “being” sustainable in the long term, the GER is mainly still focusing on ESD1, as defined by Vare and Scott (2007), because it mainly focuses on learning for sustainable development, rather than learning as sustainable development (ESD2). During the data collection sessions, some lecturers touched upon the change they observed in students. However, to assess how the students become “agents of change” thanks to education, they should be directly involved in the assessment method. This is beyond the scope of this work, and the subject of future research. In addition, the GER builds upon the existing GreenComp framework, published by the JRC. It is beyond the scope of this article to reflect on the philosophy of the GreenComp framework itself.

Finally, it was observed that after only one iteration, many lecturers started redesigning their courses. Hence, the roadmap proves to be helpful in helping lecturers reflect on their course(s) and on the program as a whole. Conducting the roadmap exercises can therefore also be considered as a tool to start conversations about integrating the competences for sustainability in a curriculum and to awaken lecturers’ creativity to do so, even in the most technical engineering courses (Wyness and Sterling, 2015). However, the exercise may not lead toward over-integration of the competences for sustainability, risking losing the topical knowledge and skills needed to obtain the learning objectives of the curriculum.

Besides topical knowledge and skills, learners should acquire sustainability competence to be able to tackle the sustainability challenges of the 21st century. The GreenComp framework identifies 12 competences for sustainability as common ground for learners and educators in the context of SE. One of its intended uses is self-assessment and reflection on and the review of curricula. The GER presented in this article, is a mixed-method approach consisting of a simultaneous hybrid data collection through semistructured interviews guided by a radar chart-based rating of the learner competences for sustainability for each course and a focus group. This article describes the step-by-step approach in detail and provides the necessary tools in its annexes for other higher education institutes to use the roadmap. Applying the roadmap to the case study of the SISSTEM bachelor program at the University of Aruba, the authors found that it not only leads to an in-depth evaluation of the curriculum, but also enables the identification of competence gaps. Second, the authors have observed that after one iteration, the exercise led to reflection amongst the lecturers involved which in turn led to an increased focus on integrating competences for sustainability in the curriculum. Conducting the roadmap exercises can therefore also be considered as a tool to start conversations about integrating the competences for sustainability in a curriculum and to awaken lecturers’ creativity to do so, even in the most technical engineering courses. It is believed that the GER is a useful tool for institutes of higher education to self-evaluate the integration of the 12 competences for sustainability in their curricula.

Ethical considerations and approval of the study: The research was submitted for privacy and ethical review (PRET) at the Social and Societal Ethics Committee of KU Leuven (file number G-2022-6079-R3(MIN). The file received a favorable opinion and was accepted on the 24th of January 2023.

This research was conducted with the financial support of the European Union through the 11th European Development Fund (FED/2019/406-549) for the Sustainable Islands Solutions through Science, Technology, Engineering, and Mathematics (SISSTEM) project. Its contents are the sole responsibility of the authors and do not necessarily reflect the views of the European Union. The authors would like to thank the interviewees for providing inputs and valuable insights into the issues discussed.

The supplementary material for this article can be found online.