Concept mapping and evaluation of the Clinical Scholars healthcare leadership program: understanding what makes health a shared value

In recent decades, there has been increased investment in leadership training programs in the medical and public health fields (Betancourt, Tan-McGrory, Kenst, Hoai Phan, & Lopez, 2017; Gilfoyle, Gottesman, & Razack, 2007; Stoller, 2013; Umble, Baker, & Woltring, 2011). Indeed, formal leadership training has been identified as an educational priority for medical professionals, beginning as early as undergraduate, medical, and residency programs (Neeley, Clyne, & Resnick-Ault, 2017; Saito, Patel, Herrera, & Kroner, 2017). These leadership training programs have engaged the health workforce in clinical, academic, and government settings to develop the skills needed to meet today’s complex public health challenges (Fernandez, Noble, Jensen, & Steffan, 2015; Fitzpatrick, Modic, Van Dyk, & Hancock, 2016; Saleh, Williams, & Balougan, 2004; Stoller, 2009), and many healthcare institutions are now allocating significant funding toward leadership programs (Straus, Soobiah, & Levinson, 2013). Benefits of effective leadership training programs have included increased participant knowledge, real-life use and application of new skills, improved communication and leadership practices, and career opportunities and/or advancement shortly after program completion (Dannels et al., 2008; Fernandez et al., 2015; Fernandez, Noble, Jensen, & Chapin, 2016; McNally and Lukens, 2006; Rosenberg, Margolis, Umble, & Chewning, 2015; Saleh et al., 2004; Straus et al., 2013).

Evaluation data are critical to understanding programmatic benefit and value to participants in medical and public health leadership programs as well as for improving program curricula for the next generation of workforce and leadership development (Straus et al., 2013). However, leadership program evaluations have faced several limitations resulting from a lack of rigor in design and implementation. These limitations include over-reliance on self-reported competency measures (often via pre-post surveys), satisfaction measures, or measures for low-level outcomes (Dannels et al., 2008; Dave, Noble, Chandler, Corbie-Smith, & Fernandez, 2021; Frich, Brewster, Cherlin, & Bradley, 2015; Harvard Business Publishing, 2018); minimal or nonexistent use of advanced training methods (e.g. real-time feedback and interactive learning; Frich et al., 2015); poor timing of impact assessment (Abrell, Rowold, Weibler, & Moenninghoof, 2011; Russon & Reinelt, 2004; Straus et al., 2013); poor or differential response rates among participants (Bergman, Fransson-Sellgren, Wahlstro¨m, & Sandahl, 2009; Dannels et al., 2008) and data that are biased or otherwise do not accurately capture experiences from all participants (Abrell et al., 2011; Dannels et al., 2008; Love et al., 2016). Given these concerns, there has been an increased demand for multi-method, community-centric evaluations of leadership training programs (Njah et al., 2021).

The Clinical Scholars (CS) National Leadership Institute was initiated as a three-year leadership development program to engage interdisciplinary healthcare teams across the US working to tackle complex health challenges in their communities (Clinical Scholars, 2020; Corbie et al., 2022; Corbie-Smith, Brandert, Fernandez, & Noble, 2022; Fernandez et al., 2022a; b; Henry et al., 2022). Funded by the Robert Wood Johnson Foundation (RWJF), the CS program centered the concept of “building a Culture of Health” (COH), which more broadly offered system change recommendations and programs to ensure attainability of optimal health for all people in the US (Dave et al., 2021; RWJF, 2020a, b). Healthcare professionals entered the CS program in cohorts (e.g. cohort 1 [2016–2019], cohort 2 [2017–2020]) and within each cohort, teams of up to five individuals implemented a real-world project that addressed a health equity issue in their home community. While participating, each cohort received (1) tailored executive coaching; (2) evidence-based leadership tools; (3) intensive leadership training via biannual in-person skill-building retreats (retreats were virtual during the COVID pandemic); and (4) a robust and customizable virtual education curriculum. The program duration allowed sufficient time for participants to work on real-world projects with measurable impact in their communities.

The evaluation plan was comprehensive and used a mixed-method approach to evaluate the effectiveness of CS leadership trainings and programmatic impact at the participant, organizational and community levels (Figure 1) (Dave et al., 2021). The evaluation was guided by Kirkpatrick’s four-level (i.e. reaction, learning, behavior, and results) model (Kirkpatrick, 1994, 2006a, b), as applicable to the CS program (i.e. individual, community and program-level outcomes) (Henry et al., 2023). Concept mapping is identified as an evaluation tool specific to level 4 of Kirkpatrick’s model (1994) and has emerged as a meaningful way to engage stakeholders in program planning and evaluation through the development of a shared conceptual framework (Trochim, 1989). A highly participatory process, concept mapping has been used as a public health research method to increase community engagement (Burke et al., 2005; Dave et al., 2018; Thompson & Burke, 2020), determine if programmatic goals are being met (Galvin, 1989), refine curricula as needed (Dave et al., 2021), and validate evaluation models (Hargett et al., 2017).

Contributing to insights of the larger CS program evaluation, concept mapping was conducted for multiple case studies (Dave et al., 2021). The case study evaluated and presented here was conducted with the second cohort of CS scholars with a focus on “making health a shared value” –a core tenet of the CS program. The purpose of this evaluation study was to (1) gain a richer understanding of how health leaders representing a broad range of disciplines collectively envision health as a shared value; (2) allow scholars to create a shared understanding of nuanced ideas beyond individual-level professional development; and (3) provide an additional evaluation data point in the assessment of long-term programmatic outcomes.

The current study focuses on the application of concept mapping to evaluate the long-term programmatic impacts at the community and organization levels corresponding to the Kirkpatrick model’s fourth level of evaluation (Kirkpatrick, 1994, 2006a, b). Concept mapping is a mixed-methods approach that involves a series of steps to collect both individual- and group-level data (Trochim, 1989). These data are then “mapped” to create a visual representation of the group’s consensus on perceptions of a topic. The resultant map depicts key themes important to the topic at hand as well as relationships between themes. Concept mapping can help evaluators develop an evaluation framework or point them toward specific measures to optimize data collection. A major strength of the approach is its meaningful engagement of participants in not just the quantitatively oriented data collection steps but also through the participatory interpretation and analysis of results (i.e. concept maps) within qualitative group discussions – ensuring the results accurately reflect participant perspectives (Burke et al., 2005).

Cohort 2 scholars were invited to participate in the concept mapping evaluation activity at the beginning of their third and final year in the CS program, at which point they had time to meaningfully engage with the program content and grow interpersonally and professionally. To avoid the common challenge of low response rates in leadership programmatic evaluations (Bergman et al., 2009; Umble et al., 2011), invitations were emailed while scholars were still in the CS program (versus post-graduation). Scholars were told they would be participating in a four-step process involving brainstorming, sorting, rating, and a group interpretation session. The first three steps took place virtually using the Concept Systems GroupWisdom online platform (Concept Systems Incorporated, 2020). The interpretation session was initially planned to occur in person at the annual spring 2020 CS retreat; however, the spring 2020 retreat was conducted virtually due to the COVID-19 pandemic (Fernandez et al., 2021).

The focal prompt for this study asked scholars to conceptualize what it means to “make health a shared value;” it was developed by the CS evaluation team and integrated into the concept mapping software (i.e. sorting cards and prompts), as there was no predetermined or validated instrument to begin with or survey to field. During the sorting and rating activities, the evaluation team administered a brief electronic survey to collect scholars’ demographic information (i.e. age, gender, sector of employment, academic/professional discipline and geographic location). Scholars were given three weeks to asynchronously complete the online activities, with the expectation that each activity would take approximately 20 min to complete. All participants consented prior to completing study activities and were informed that a lack of participation would not impact their status as a scholar. All ethical IRB principles at the [institution anonymized for peer review] were followed. To build potential future capacity for utilizing concept mapping, scholars participating in this study also received a one-hour training focused on concept mapping research skills.

• Step 1: Brainstorming

Upon starting the brainstorming activity, scholars were given background information on the COH Action Framework (see RWJF, 2020b), a description of project goals and participation expectations, and instructions for how to complete the brainstorming activity. In this activity, scholars were asked to generate a list of statements (hereafter referred to as “items”) in response to the focal prompt: “What components are essential for making health a shared value?” Scholars were instructed to list all words or phrases they felt answered the prompt, and no limit was given to the number of responses they could provide. All responses were combined into a de-duplicated list to be utilized for the following sorting and rating activities.

• Step 2: Sorting and rating

Sorting. Using the GroupWisdom Software, scholars were asked to independently categorize the combined, final list of items (n = 45) into groups or piles that made sense to them by thematic similarity. Scholars then suggested names for each of their sorted piles.

Ratings of importance. All items were individually rated across three scales of importance. These scales utilized questions developed by the CS evaluation team based on drivers previously identified by RWJF to make health a shared value (RWJF, 2020a): (1) “How important is this item to shaping mindsets and expectations around health?” (referred to as “Mindsets and expectations”); (2) “How important is this item to supporting civic engagement around health?” (referred to as “Civic engagement”); and (3) “How important is this item to building a sense of community?” (referred to as “Sense of community”). All three questions utilized a five-point Likert-type response scale (1 = not important at all; 5 = extremely important).

• Step 3: Virtual discussion

The synchronous interpretation session was conducted via Zoom (Zoom, 2020) during the CS Spring 2020 virtual retreat. As part of the retreat, CS program staff held a two-hour interpretation discussion, with the first 60 min allotted to training in the concept mapping evaluation approach and the final 60 min dedicated to reporting and participatory interpretation of the results from the data scholars provided in the months prior. Specifically, scholars were provided with a description of concept mapping as a participatory method and the overall concept mapping process, and then they were shown progress on the CS evaluation project, including the final point and cluster maps. Scholars worked in small groups to discuss each cluster, answering questions such as (1) “What do these items have in common?”; (2) “Which items do you find important?”; (3) “Which items surprised you?” and (4) “What would you call this cluster?” After the small group discussion, all participants reconvened as a full group to review clusters and determine cluster names. After each small group presented name ideas, the large group voted on final cluster names.

The CS evaluation team partnered with concept mapping experts at the University of Pittsburgh Concept Mapping Institute (https://www.sph.pitt.edu/bchs/research-and-practice/concept-mapping-institute) to conduct analyses of the sorting and rating data. Using the GroupWisdom platform, all scholar sorting data were combined into a single dataset. First, using similarity matrices (i.e. a combined N x N binary matrix of sorted item similarities) and non-metric multidimensional scaling, a spatial point map was created. This point map displays item relationships in a two-dimensional (x, y) plane, where relative distance between the items reflects perceived similarity among scholars. Then, hierarchical cluster analysis was used to determine how the items clustered together. This analysis allowed for determination of the best cluster solution by partitioning the data based on group consensus, which is shown in a cluster map. Final versions of these point and cluster maps were shared with scholars at the interpretation session (Figure 2). The authors also created various pattern matches or visual displays of the relationship between mean cluster ratings across two variables. Additionally, bi-variate “Go-Zone” plots were generated to compare differences of individual item ratings. Both the pattern matches and “Go-Zone” plots use Pearson correlations to assess comparisons between variables.

This evaluation included scholars who began the CS program in 2017 and were in their third and final year of program participation (N = 33). Of the 33 scholars invited to participate, 26 (78.7%) completed the brainstorming activity, 18 (54.5%) completed the sorting activity, 16 (48.4%) completed the rating activity and 33 (100.0%) scholars participated in the virtual discussion. Of the scholars who provided demographic data (n = 23) (Table 1), 52.2% worked in healthcare and 39.1% worked in academia. Most scholars identified as female (78.3%) and worked in urban settings (60.9%), followed by rural (21.7%) and suburban (17.4%) settings. Over 72% of scholars conducted research or clinical practice within the areas of medicine, psychology, or social work. While we did not collect data on career level for only those who completed this concept mapping study, 50.0% of scholars across the entire cohort 2 were mid-career level (i.e. 6–14 years of experience), 39.5% were at an advanced career level (i.e. fifteen or more years of experience) and 10.5% were early career (i.e. five years or fewer). Additionally, more than half of scholars in cohort 2 were physicians or nurse practitioners (54.9%).

Seventy-five items were proposed by scholars during the initial brainstorming activity and after removing all duplicates, 45 unique items remained. The generated point map of these items had a stress value of 0.285, indicating a good fit of item content to emerging themes identified. The evaluation team determined a four-cluster solution best fit the data, reflecting the mean number of clusters made by participants in the sorting activity (3.11 [range: 3–6]). Tables 2–5 show the 45 items in their respective clusters along with the mean item ratings for the three areas of importance rating scales: Mindsets and expectations, Civic engagement, and Sense of community. There were similar average cluster ratings for each rating scale with high correlations between rating scales. Additionally, Figure 2 shows the combined point and cluster map, including the final four-cluster solution with participant-generated names.

Cluster 1, named Shared Understanding and Awareness of Health, consisted of 11 items (Table 2). Three items were rated highly across at least two of the three scales: item #1 – clear communication; item #25 – defining health in a more comprehensive way that considers elements beyond physical health (e.g. social needs, mental health, and substance use); and item #10 – understanding different perspectives that drive health and wellness.

Cluster 2, named Big Picture Strategies and Growth Mindset, consisted of six items (Table 3). Item #22 was rated highly across at least two of the three scales: “being open to innovative thinking.”

Cluster 3, Collaboration and Community Engagement, consisted of 14 items (Table 4). Five items were rated highly across at least two of the three scales: item # 37 – willingness to collaborate; item # 28 – connecting health with the community; item #2 – establishment of rapport; item # 23 – collaboration with community; and item #29 – effective community partnerships (e.g. with community-based organizations and health plan partners).

Cluster 4, Health Equity and Inclusion, consisted of 14 items (Table 5). Three items were rated highly across at least two of the three scales: item #12 – accessibility, equality and equity across all populations; item #27 – basic health needs (e.g. safe housing, accessible fresh food and access to clean water) and item #41 – time and resources.

Across all clusters, specific items were rated highly for one or more areas of importance (i.e. mindsets and expectations, civic engagement, and sense of community; Table 6). When assessing the relationship between the three rating scales focused on various areas of importance, all had high correlations (r = 0.86 to 0.97). The greatest difference between scales existed between the Mindsets and expectations scale and Sense of community scale; thus, Figure 4 depicts a “Go-Zone” graph using these scales. Additionally, Figure 3 shows the relationship between the Mindsets and expectations and Sense of community rating scales. For all scales, Cluster 3 (Collaboration and Community Engagement) had the highest rating. A “Go-Zone” comparing the ratings for the items in this Cluster 3 across the Mindsets and expectations and the Sense of community scales (Figure 4) indicated less consensus on which items held the most importance (r = 0.32); those in the green zone of consensus (i.e. “Go-Zone”) include items: (1) establishments of rapport; (2) collaboration with community; (3) connecting health with the community; (4) effective community partnerships and (5) willingness to collaborate.

To our knowledge, this study is one of the first of its kind to use concept mapping as part of a larger evaluation of a multi-year, equity-focused leadership training program. Specifically, this study captured key elements needed for long-term organizational and systemic change in an effort to make health a shared value, from the perspective of CS participants. The four clusters (i.e. Shared Understanding and Awareness of Health, Big Picture Strategies and Growth Mindset, Collaboration and Community Engagement, and Health Equity and Inclusion) and 45 unique items developed by scholars shed light on a wide range of important perspectives and were formed via an iterative, informed analysis of these perspectives. Each cluster had at least one item that scholars rated as highly important based on the three scales pertaining to RWJF-reported drivers essential for making health a shared value. Items developed by scholars that received lower rankings of importance could be explored further to see if they are important in other ways not reflected by the three rating scales used in this study. Results from this evaluation study supported the study objectives of understanding how scholars envision health as a shared value (beyond individual-level professional development) and creating a visual interpretation of that understanding. Additionally, these results will be used as an additional evaluation data point in the long-term assessment of CS programmatic outcomes (e.g. understanding how the extant program curricula maps onto these clusters, identifying areas to expand or adapt future program curricula, and lessons learned for future concept mapping evaluations with CS scholars).

The resulting areas from the concept map have implications for improvement within the health education system and continued professional education opportunities if progress is to be made toward making health a shared value in the US. Despite the obvious importance of making health a shared value, the concept itself is highly complex and challenging to articulate and conceptualize. The results of this evaluation study identify and clarify the critical areas for action and adaptation of curricula for future and current health professionals (e.g. clear communication to foster a shared understanding and awareness of health; being open to innovative thinking to promote big-picture strategies and a growth mindset). Further, programs aiming to help health professionals make health a shared value may find it useful to focus programmatic curricula and training efforts on items within Cluster 3 (“Collaboration and Community Engagement”) – the highest-rated cluster among areas of importance – and/or on specific items across clusters that rated highly on all three rating scales for areas of importance [i.e. item #’s 1 (Clear communication), 25 (Defining health in a more comprehensive way that considers elements beyond physical health (e.g. social needs, mental health, and substance use), 22 (Being open to innovative thinking), 12 (Accessibility, equality, and equity across all populations), and 41 (Time and resources)]. Additionally, programs might consider how each area of importance impacts items across clusters; civic engagement, for example, has the most “top items” (Table 6) compared to mindsets and expectations and sense of community. Lastly, these findings could be of further interest beyond the health field, especially considering the ways in which community engagement and participation were reflected in the findings and the integrated nature of environment, social determinants, and health outcomes.

For similar evaluations of healthcare leadership training programs, concept mapping should be considered as one component of a multi-method and long-term evaluation plan. In addition to other analyses that assess programmatic value and benefit from the perspective of program leaders, funders, or other high-level stakeholders, concept mapping provides participatory analysis that identifies mindsets and what matters to participants. This participatory assessment can be particularly helpful to evaluators outside of the program implementation when assessing program impact. Indeed, participatory and community-based evaluation methods have been identified as critical in the evaluation of leadership training programs (Njah et al., 2021). To ensure participants can thoughtfully reflect on the program, we suggest incorporating concept mapping at the later stages of program participation. Concept mapping prompts should follow guidance set by Trochim (1989) (i.e. developing the focal prompt for brainstorming, developing the focus for rating of items), and should center programmatic goals and areas identified as important by funders or other stakeholders. Additionally, this concept mapping evaluation study is part of a series of concept mapping evaluations conducted by the CS evaluation team with different cohorts of scholars and at various time points throughout the CS program (Carr et al., 2024; Dave et al., 2021); this may serve as an example of how to use concept mapping for future evaluations of healthcare leadership programs that have a similar cohort-based and multi-year structure. Lastly, future studies could explore the use of concept mapping after the conclusion of a program to assess more long-term programmatic impact.

This study has several strengths. The concept mapping methodology includes individual perspectives and group consensus in an approach that can be adapted to different-sized groups, is flexible to varying participation across sessions, and is facilitated in virtual (and largely asynchronous) environments. Scholars were not limited in the number of responses they could contribute, yielding diversity in the type and scope of concept items. As described by Trochim (1989) and compared to more traditional qualitative methods of interviewing, coding, and synthesizing of thematic findings – we found concept mapping to provide a logistical and participatory benefit as analysis was largely conducted by the scholars themselves (versus the evaluation team) through facilitated group and individual reflection. Additionally, the concept mapping approach also includes a quantitative measure of connectedness across items and the ability to plot areas spatially, offering a novel visual framework of related groupings and concepts.

There are limitations that should be considered alongside these strengths. First, the participant sample was fairly homogenous, with the majority of scholars identifying as female (78.3%) and working in healthcare (52.2%). However, the purposes of this study were not to provide highly generalizable data, but rather to understand perspectives from a very specific group and provide data to include in the larger, multi-method evaluation of the CS program. Second, only scholars in their third year of the CS program were invited to participate instead of including all existing cohorts in the study, which means there are likely additional perspectives from CS scholars that were not included in this analysis (i.e. scholars in their first or second year of the program, as well as scholars who had already graduated). However, we only recruited scholars in their third year (i.e. cohort 2) of the program because we wanted to ensure participants had nearly finished the program, had time to meaningfully engage with and reflect on the program, and had time to advance as healthcare leaders and/or employ their skills learned during CS prior to answering the prompt. Third, given that much of the analyses were finalized via a virtual synchronous discussion, it is possible certain perspectives were over- and underrepresented due to the inherent limitations of group work (e.g. “louder” voices monopolizing the discussion). We attempted to control for this by providing guiding questions for group time and coming back as a large group with a trained facilitator to support a collaborative and equitable conversation. Additionally, as this concept mapping evaluation was focused solely on cohort 2, this may have further facilitated group discussion, as the scholars were familiar with one another and had been participating in the program together for three years. Fourth, our discussion/interpretation session took place virtually rather than in-person, due to the COVID-19 pandemic, which could have impacted group dynamics and collaboration. Notably, while not all scholars completed each phase of the study (i.e. brainstorming, sorting, rating and in-person discussion), this is not considered a limitation, as concept mapping is structured to allow for different perspectives within different steps and overall aims for group consensus.

As a participatory and iterative approach with the CS 2017 cohort, this concept mapping study successfully centered participant perspectives as data to comprehensively answer our focal prompt – one of the program’s core tenets – “making health a shared value.” Via a conceptual and visual understanding of specific factors, as well as intra- and inter-cluster relationships, the shared concepts mapped through this study clearly depict the consensus-identified importance of four primary clusters: (1) Shared Understanding and Awareness of Health, (2) Big Picture Strategies and Growth Mindset, (3) Collaboration and Community Engagement and (4) Health Equity and Inclusion. As one part of a multi-method evaluation plan, these findings provide support of the CS program’s value to participants, as evidenced by their shared understanding regarding what makes health a shared value and the similar average cluster ratings for each rating scale, developed based on RWJF-identified drivers of importance, with high correlations between all rating scales.

The results from this concept mapping study provide a framework for the societal value of health in the US as well as for the smaller-scale evaluation of a health research and leadership program. Future research should incorporate concept mapping into the long-term evaluation of other healthcare leadership training programs to understand its utility in other settings and ensure a participant-centered and participatory evaluation. These findings may be of use to targeted training programs, health systems, or education programs seeking to advance research opportunities as well as to community-based programs looking to clearly depict progress toward making health a shared value.

The authors acknowledge the participation of the Clinical Scholars Fellows in this concept mapping study and the program overall. The authors would also like to thank Lia Garman for her contributions during the early stages of this work.

Funding: Clinical Scholars was funded by the Robert Wood Johnson Foundation.