A design science approach to developing and evaluating items for the assessment of transversal professional competences

The validation of prior learning (VPL) is defined as the process of “assessing and recognising a wide range of skills and competences which people develop through their lives and in different contexts, for example through education, work and leisure activities” (Bjørnåvold, 2000b, p. 216) and plays an increasingly important role in the educational policies of the European Union (EU) and beyond. It is supported by the decision to foster lifelong learning (Council of the European Union, 2006), through the introduction of the European Qualifications Framework (EQF) (Council of the European Union, 2017) and the recommendation to validate non-formal and informal learning regardless of the route of acquisition (Council of the European Union, 2012). Developing viable and efficient methods for the assessment of professional competences in VPL procedures may facilitate a decrease in unemployment, increase labour market mobility and facilitate social cohesion and social justice in the EU (Harris, 1999; Jackson, 2011; Jarvis, 2007). While legislation for the VPL are in place in most of the EU members states, developing innovative and standardised methods for the assessment of professional competences within VPL continues to challenge EU policy-makers (Council of the European Union, 2012, 2017) and researchers (Bohne et al., 2017; Brockmann et al., 2009).

As we currently lack innovative and standardised approaches for the assessment of professional competences in the VPL (Cedefop, 2017, p. 20), the EU calls for a “standardisation of tools and the use of information and communication technology (ICT)” (Cedefop, 2017, p. 20) to support VPL as its use is currently “not widespread” (Cedefop, 2017, p. 72). Furthermore, the EQF, a meta framework to translate qualifications from one country to another, is often considered to be the “lowest common denominator” on which the EU member states could agree upon and thus lacks the breadth and depths of other taxonomies, such as the Occupational Information Network (O*NET), to describe professions (Markowitsch and Luomi-Messerer, 2007, p. 50). In addition, this paper is practically motivated by a project conducted in cooperation with the Austrian Federal Economic Chamber aiming to design an ICT artefact to support the VPL. To do so, a previously proposed model of professional competences (Fahrenbach et al., 2019a, b) is complemented by a self-assessment for transversal professional competences. Consequently, the research question of this paper is “How can we develop and evaluate candidate items for the assessment of transversal professional competences in an ICT artefact?”

In this paper, we draw on a design science research paradigm (Gregor and Hevner, 2013; Hevner et al., 2004; Peffers et al., 2007) to introduce an ICT artefact supporting the assessment of transversal professional competences in the VPL. The purpose of this artefact is to facilitate VPL procedures for 80 professions in the Austrian trade and craft sector. The design science paradigm scaffolds the development (section 5.1) and evaluation of candidate items (section 5.2). We describe the development of candidate items (in the German language) through a qualitative content analysis (Krippendorff, 2009; Mayring, 2015) of the qualification standards of Austrian trade and craft professions (section 5.1.1). To further refine the quality of these candidate items, we identified critical takeholders (Achterkamp and Vos, 2007) (section 5.1.2) and conducted 15 focus groups (Recker, 2013) (section 5.1.3). To evaluate the development process as a whole, we conducted two expert interviews (Bogner and Menz, 2016; Brinkmann, 2013) (section 5.2.1). To evaluate how well pupils understand the items, we tested the assignment between candidate items and the hierarchical taxonomy and in workshops in two vocational schools (section 5.2.2). To further evaluate how pupils perceive the usability of the ICT artefact, we conducted a usability texting in two additional vocational schools (Rubin and Chisnell, 2008) (section 5.2.3). A last phase of evaluation describes the consolidation of the data acquired previously (section 5.2.4). Reflecting on the development and evaluation of these candidate items, we draw a conceptual bridge between the O*NET and the EQF. We describe how this hierarchical taxonomy can be assessed via the items that are developed based on five qualification standards of Austrian trade and craft professions ^[1]. Furthermore, we find that the conceptual bridge can be meaningfully assessed through 160 items. Practically, we present a use case of an ICT artefact that is intended to cover the whole VPL procedure for 80 different professions in the Austrian trade and craft sector, thus filling the gap to put specific methods targeting the VPL into practice.

The remainder of this paper is structured as follows. Section 2 provides a literature review. Section 3 positions the work in a design science research paradigm, as it provides a comprehensive framework to guide development and research activities. Section 4 introduces the motivation behind this research endeavour and outlines objectives a solution should fulfil. Section 5 presents the main part of the paper. Section 6 outlines practical implications. Section 7 discusses limitations, opportunities for further research and concludes.

The notion of professional competences has a long history. What makes a person a competent practitioner is already discussed since Plato (Mulder and Winterton, 2017; Sandberg and Pinnington, 2009). In the recent history, competence has been first mentioned by White (1959) as a personality trait underlying motivation. It was taken up in the psychological discourse as an alternative to test a candidate's intelligence (McClelland, 1973) and as linguistic competence (Chomsky, 1965) to describe the often tacit knowledge to creatively deal with language.

Professional competences and their assessment are relevant in several academic disciplines such as medicine (Epstein, 2002; Epstein et al., 2007), nursing (Girot, 1993), vocational education and training (VET) (Bohne et al., 2017; Mulder et al., 2007), human resources management (Boyatzis, 1982), management (Sandberg, 2000; Sandberg and Pinnington, 2009; Sandberg and Targama, 2007) and more practically oriented discourses such as lifelong learning or the VPL (Cedefop, 2015, 2017).

Different regions have their own tradition in defining professional competences (Le Deist and Winterton, 2005; Mulder et al., 2007). The American tradition usually defined professional competences from a behavioural perspective as the knowledge, skills, abilities and other characteristics underlying superior performance (Boyatzis, 1982; Spencer and Spencer, 1993).

Scholars in the United Kingdom followed a more functionalist approach relying on specific performance criteria (Le Deist and Winterton, 2005) and, based on this reasoning, policy-makers developed National Vocational Qualifications. Especially since the advent of higher and degree apprenticeships in the United Kingdom, professional competences are more frequently in the focus of policy-makers (Bravenboer and Lester, 2016; Lester, 2017; Lester and Bravenboer, 2020; Lester et al., 2016). However, regardless of its international application, the functional analysis of outcome-based VET qualifications has been criticised by many scholars (Grugulis, 2006; Young and Diem, 2016).

A more holistic approach was adopted in German-speaking regions (Le Deist and Winterton, 2005), in which vocational action competence (“Handlungskompetenz”) is defined in terms of personal competence, social competence, method competence and domain competence. This understanding of professional competence is also used in this paper.

Conceptually, the notion of professional competence can be distinguished in entity-based and relational perspectives. So-called “entity-based perspectives” (Sandberg and Pinnington, 2009) describe professional competences as an “internal, attributes-based” (Lester, 2014, p. 39) property of the individual. So-called “relational perspectives” (Sandberg and Pinnington, 2009) see professional competences as external to the individual and based in the activity of a person (Lester, 2014, p. 40). This perspective sees professional competence as “knowing-in-action” (Schön, 1983, 1990; Sennett, 2008), determined by a person's understanding of work (Sandberg, 2000), embedded within practice (Lave, 2010; Lave and Wenger, 1991), activity (Engeström et al., 1999) or as “skilful performance” (Sandberg et al., 2017).

When we consider the assessment of transversal professional competences, we have to consider three distinct yet intertwined dimensions (Bergsmann et al., 2015, p. 3). We first outline levels of professional competences, second a content dimension of professional competences and third, competence development. Figure 1 gives an exemplary overview on these three dimensions.

First, several frameworks are published to describe levels of professional competence that state the degree of expertise or professional competence verbally and/or numerical. These can be distinguished in policy-pragmatic approaches and theoretical/research based approaches (Bergsmann et al., 2018). Policy-pragmatic approaches include for example the EQF (Council of the European Union, 2017), the SEEC level taxonomy (SEEC, 2016, 2019) or the International Standard Classification of Education (UNESCO, 2011). Scientific theoretical approaches include Dreyfus' model of skill acquisition (Dreyfus and Dreyfus, 1986), Bloom's taxonomy of learning outcomes (Anderson, 2001; Bloom et al., 1956) and the structure of the observed learning outcomes (Biggs and Collis, 1982). For the purpose of this paper, we refer to the taxonomy of learning outcomes (Anderson, 2001; Bloom et al., 1956) and to the EQF. In the following, we briefly describe the EQF in more detail as it becomes established in the EU member states and in this vein, professional associations within the EU adapt their qualification standards to the EQF. This requires professional associations to develop qualification standards, which are documents aimed at making all relevant learning outcomes for a specific profession (in terms of knowledge, skills and competences) explicit (Council of the European Union, 2017). Thus, the EQF sets out to describe every profession in terms of their specific knowledge, skills and competences, whereas competences are to be understood as autonomy and responsibility. The EQF is designed as a translation device that aims at making national qualifications more comparable across the EU. The EQF is criticised regularly. It is described as reductionist as it is the lowest common denominator EU member states could agree upon (Markowitsch and Luomi-Messerer, 2007, p. 50). In particular the EQF but qualification frameworks in general are subject to regular criticism (Allais, 2011; Guillen et al., 2007).

Second, the content dimension of professional competences are usually referred to as competence areas, competence standards or competences frameworks that describe the “what” in a competence model based on underlying psychological and work-related constructs (Le Deist and Winterton, 2005; Lester, 2014). This paper relies on a recently introduced content model (Fahrenbach et al., 2019a, b) which is based on the O*NET ^[2]. The O*NET has been developed in America over a considerable amount of time. It replaces the Dictionary of Occupational Titles and is a comprehensive model of occupational characteristics and requirements (Council of the European Union, 2017). The O*NET has been developed in a Fordist and Taylorist tradition and aims at describing characteristics and requirements in precise detail (Burrus et al., 2013) to make them assessable. The O*NET is not a theory, rather it is a “comprehensive system designed to describe occupations” (Peterson et al., 2001, p. 451) relying on a hierarchical taxonomy (Peterson et al., 2001, p. 452). This hierarchical content model distinguishes worker oriented descriptors (worker characteristics, worker requirements, experience requirements), job oriented descriptors (occupational requirements, workforce characteristics and occupation-specific information) that can be cross-occupational (worker characteristics, occupational requirements) and occupation-specific (experience requirements, occupation-specific information). Beneath these high-level descriptors, it contains more than 500 even more detailed occupational descriptors (Burrus et al., 2013). The O*NET impressively demonstrates that “a precise description of occupations and jobs requires more dimensions than knowledge, skills and competence, and makes the EQF's reductionist approach to qualifications clear” (Markowitsch and Luomi-Messerer, 2007, p. 50). In this regard, the EQF perspective could benefit from the depth and breadth that the O*NET offers.

Third, competence development departs from the assumption that professional competences are not static but can be enhanced or developed through education and training (Bergsmann et al., 2015, p. 3). For example, one would expect that a person holding a qualification on EQF level 6 (in VET: master craftsmen) are able to perform on a higher level of proficiency than a person holding a degree on EQF level 4 (apprenticeship).

The assessment of professional competences is subject to a vivid debate (Colardyn and Bjornavold, 2004). Usually, scholars distinguish between formative and summative assessment. In this context, formative assessment can be summarised as collecting evidence that a person has acquired certain professional competences, usually through portfolios or and supported by a guide or facilitator. Summative assessment employs testing and psychometric methods to determine whether people have acquired certain professional competences. The “gold standard” is a combination of several assessment methods (Cedefop, 2017, p. 72).

Methodologically, the development and evaluation of the ICT artefact rests on a design science research paradigm (Gregor and Hevner, 2013; Hevner et al., 2004; Peffers et al., 2007). Design science “attempts to create things that serve human purposes” (Simon, 1969, p. 55), while natural sciences and social sciences seek to understand reality. Design science addresses the “creation and evaluation of an innovative and purposeful artefact for a specified, currently unresolved problem domain” (see Hevner et al., 2004, p. 82). As utility is its primary goal, design science addresses research problems through the “building and evaluation of artefacts designed to meet the identified […] need” (Hevner et al., 2004, pp. 79–80). Within a design science research paradigm, an artefact is a thing that “has, or can be transformed into, a material existence as an artificially made object (e.g. model, instantiation) or process (e.g. method, software)” (Gregor and Hevner, 2013). Commonly, the design science process includes six subsequent phases: (1) problem identification and motivation, (2) definition of the objectives for a solution, (3) design and development, (4) demonstration, (5) evaluation and (6) communication” (Peffers et al., 2007, p. 46). Design science research is concerned with methodological rigour that is achieved by “appropriately applying existing foundations and methodologies” (Hevner et al., 2004, p. 80). Referring to the six phases outlined above, we describe the development of candidate items in German (section 5.1) and their subsequent evaluation (section 5.2).

Corresponding to the second and third phase of the design science research paradigm, this section describes problem identification, motivation and the requirements a viable solution should fulfil. The present study is practically motivated by the current situation of the Austrian trade and craft sector. All companies in the trade and craft sector are represented by 80 professional associations federally represented by the Austrian Federal Economic Chamber. First, as with most other European countries, the Austrian trade and craft sector faces a shortage of skilled labour (Brixiova et al., 2009) as more people forego a career in the trade and craft sector in pursuit of a tertiary education. Second, supporting the integration of the labour market in the EU, the Austrian trade and craft sector aligns their qualifications to the National Qualifications Framework (NQF) and the EQF. In line with the introduction of the NQF and the EQF, one can witness a shift from input-oriented education towards output-oriented education in VET within the Austrian trade and craft sector (Spady, 1994). Indicating that if people demonstrate the necessary knowledge, skills and competences to perform a profession according to a certain standard, they should be given the right to do so, regardless of how they acquired these knowledge, skills and competences.

One is allowed to open a business in Austria if they can prove a qualification on NQF and EQF level 6. How level 6 is defined is written in greater detail in the respective qualification standard. These qualification standards are currently developed by the Austrian trade and craft sector for each single profession. As output-oriented education gains increasing importance, we also witness a shift towards competences and learning outcomes in the VET in the Austrian trade and craft sector. VET schools shift their emphasis from defining (factual) knowledge that serves as the input for education towards defining learning outcomes which describe what a person is able to do after completing a course (Biemans et al., 2009).

As these changes take place, the Austrian trade and craft sector faces a dilemma. There is a need to foster entrepreneurship (i.e. making it easier for people to open a business as long as they can prove the necessary knowledge, skills and competences) to fill the shortage of skilled labour. However, maintaining the high quality of practice and professionalism of the workforce within the trade and craft sector continues to be a challenge. To tackle the problems described above, the trade and craft sector represented by the Austrian Federal Economic Chamber initiated a research project to design an abstract (i.e. not specific to a certain profession) and innovative (i.e. ICT-based) artefact to support the whole VPL procedure focussing on the assessment of transversal professional competences.

Based on these considerations, the designed artefact must fulfil certain requirements (Gregor and Hevner, 2013). First, the artefact needs to be in accordance with the respective European legislation (Council of the European Union, 2006, 2012, 2017). Second, the artefact should be relevant to practice. It should be applicable within the whole of Austria to ease the process of assessing applications without a formal qualification (i.e. it should make the work easier of those who assess applicants that do not provide formal learning outcomes). These requirements refer to the standards of “utility” (i.e. ensuring that the information needs of intended users and stakeholders are considered) and “feasibility” (i.e. not only researchers but also practitioners in the field can work with the artefact) (Patton, 2012, p. 5).

As the artefact should be able to incorporate the qualification standards of 80 different professions in the Austrian trade and craft sector on EQF level 6, it should be perceived as both, useful and feasible by the critical stakeholders (Achterkamp and Vos, 2007) that will interact in the ICT artefact. Third, to achieve these goals, the artefact must meet certain conceptual requirements. It should be able to deal with formal, non-formal and informal learning outcomes (Bjørnåvold, 2000a, b; Eraut, 2004). The artefact should be able to comprehensively describe professional competences (i.e. it must be applicable to 80 different professions) (Peterson et al., 2001). The artefact should consist of a transversal (Le Deist and Winterton, 2005) component and a component specific to a certain profession. Furthermore, the artefact should be able to both describe levels of competence and individual competence acquisition (Anderson, 2001).

Based on legal, practical and conceptual requirements, an artefact should be an ICT artefact (Gregor and Hevner, 2013) which supports the whole VPL procedure (Bjørnåvold, 2000a, b), usually consisting of the identification, documentation, assessment and recognition of prior learning. For identification of professional competences, the artefact should offer adequate guidance. Likewise for the documentation, the artefact should support the upload of relevant documents to prove learning outcomes. The artefact should support different methods of assessment (i.e. represent the gold standard in competence assessment (Cedefop, 2017, p. 72)), but at the very least, it should be comprised of a self- and other-assessment. Lastly, it should present assessment results in a way efficient enough to recognise learning outcomes and award a partial or full qualification.

This section presents the development of candidate items (section 5.1) and their evaluation (section 5.2) in more detail. The procedure was based on a common procedure to instrument development in information systems (Recker, 2013, p. 73) and adapted to fit our study design. Phases are presented step-wise to increase transparency. As a consequence, this procedure can be described as a “participatory multi-perspective, multi-stakeholder approach” (Bergsmann et al., 2018, p. 543) to develop and evaluate candidate items. Table 1 depicts the research process in further detail.

As outlined in the methods section, this section aims at describing the development of candidate items in more detail. Candidate items are short written statements of what a person is capable of doing. This section comprises of three phases that are outlined below.

This phase aimed at developing candidate items that can be used for a self-assessment of professional competences. Following common procedures of instrument development in information systems (Recker, 2013, p. 73), we developed items of transversal (i.e. not specific to a certain profession) professional competences. We first conducted a content analysis of five qualification standards of different professions in the Austrian trade and craft sector. Subsequently, we developed items in several iterations, including feedback loops within the research group and with the Austrian Federal Economic Chamber.

We conducted a content analysis – a qualitative research method that allows the capturing and structuring of the content of large bodies of text (Krippendorff, 2009; Mayring, 2015) – of five qualification standards of different Austrian professions (Plumbers, Butchers, Hairdressers, Orthopaedic Shoemakers and Motor Vehicle Technicians) that were formulated according to EQF on level 6 (Council of the European Union, 2017). A qualification standard is a document that comprehensively lists the necessary learning outcomes needed to be able to perform on a specific level of professional competence. Each qualification standard lists roughly 50 learning outcomes that are described in more detail in terms of knowledge (i.e. knowing that), skills (i.e. knowing how) and competences (i.e. autonomy and responsibility). Obtaining a qualification on level 6 allows people to open their own business in the profession and enables them to employ others. The content analysis revealed that roughly 2/3 of all learning outcomes of different professions are comparable–and thus qualify as transversal professional competences (Le Deist and Winterton, 2005). For example, the social competence to deal with customers is relevant in nearly every profession, while the skill to cut hair is relevant for hairdressers but not for butchers.

Based on the content analysis of five different qualification standards, we developed 146 candidate items (Recker, 2013, p. 73) within the research group. While one member of the group conducted the content analysis and developed candidate items, other members assessed these items and provided feedback on them (i.e. accounting for an expert panel in the item development procedure) (Recker, 2013, p. 73). During this iteration, all 146 items were reformulated with the aim of shortening them and improving their clarity. Where deemed necessary, candidate items were relocated to a more appropriate competence dimension (i.e. sub-competence) within the underlying hierarchical taxonomy. A second expert panel, consisted of four people, all members of Austrian Federal Economic Chamber's trade and craft sector, who reviewed all 146 candidate items again and adapted them where deemed necessary. All members of the second expert panel finished tertiary education and one completed a doctoral degree. Subsequently, a set of 146 candidate items served as an input for the focus groups described in section 5.1.3.

This phase aimed at identifying critical stakeholders that are able to judge the quality of a candidate item and further refine these items. First, the Austrian Federal Economic Chamber identified critical stakeholders who should take part in the focus groups evaluating the 146 candidate items. A stakeholder (by definition) of an organisation is “any group or individual who can affect or is affected by the achievement of the organisation's objectives” (Achterkamp and Vos, 2007, pp. 4–5). Following a published procedure to identify critical stakeholders in projects (Achterkamp and Vos, 2007; Mitchell et al., 1997), members of the Austrian Federal Economic Chamber identified five relevant stakeholder groups in a workshop facilitated by the research group:

1. Entrepreneurs within the trade and craft sector: Representatives of this stakeholder group own a business and work in the Austrian trade and craft sector. They are specialists in certain professions and train apprentices.

2. Educational politicians: Representatives of educational politics work in governmental organisations and handle educational questions.

3. Teachers in VET schools: Representatives of this stakeholder group can estimate the knowledge, skills and competences of their pupils. They are able to assess whether an item is adequately difficult and can be understood by pupils.

4. Educational scientists: Representatives of this stakeholder group are acquainted with the formulation of competences and learning outcomes in general.

5. Representatives of professional associations: Representatives of this stakeholder group usually develop the qualification standards for a certain profession. Furthermore, they are in charge of the VPL in the Austrian trade and craft sector.

This phase aimed at ensuring that each candidate item can be (linguistically) understood and reflects the practical experience of all stakeholder groups involved. Furthermore, the research group was able to gather feedback from practitioners in the field. In sum, we conducted 15 focus groups in six different cities in Austria between 10/14/2018 and 12/11/2018. Each focus group lasted approximately three hours, was moderated by a member of the research group and hosted by the Economic Chamber in the respective city. In total, 73 people (55 male, 18 female) participated in the focus group discussions. After asking for consent, we recorded each focus group.

The total sample consisted of 15 entrepreneurs within the trade and craft sector (11 male, 4 female), 14 representatives of educational politics (11 male, 3 female), 15 teachers in VET schools (11 male, 4 female), 15 educational scientists (11 male, 4 female) and 15 representatives of professional associations (10 male, 5 female). All participants within the sample completed at least VET. Sixty-four participants obtained a degree from a tertiary institution (Bachelor, master or equivalent), sixteen people obtained at least a doctoral degree and four a post-doctoral degree.

In each focus group, we went through 10 candidate items and proposed verbs (Bloom et al., 1956; Dreyfus and Dreyfus, 1986; Krathwohl, 2002) indicating the level of competence for each candidate item. Item-verb combinations are intended to be a form of self-assessment for candidates where they can choose the most appropriate verb for their estimated level of professional competence in regard to the item. While the item determines the content dimension, a chosen verb determines the level of competence in regard to the content.

During the focus groups, we asked the following questions for each candidate item: First, Do you understand the candidate item? If at least two participants did not understand the item or thought it to be too complicated, the group reformulated the item till a sufficient wording was found while maintaining the integrity of its meaning as much as possible. Second, we presented the focus groups a set of two to four verbs for each of the six competence levels and each candidate item. We asked for each verb: Does the verb fit the candidate item? And does the verb reflect the respective level of competence?.

We give an example here: First, the candidate item “I am able to … . … the efficiency of processes in my business” was put up for discussion and was reformulated if deemed necessary. Second, we presented several verbs on each level of competence that fit to the candidate item (e.g. Level 1: to recognise, to name; Level 2: to explain, to describe; Level 3: to interpret, to assess; Level 4: to analyse, to test; Level 5: to argue and justify, to judge; Level 6: to optimise, to evaluate). Participants then evaluated (1) whether the proposed verbs fit to the candidate item and (2) whether the verbs are sufficiently distinct on each level of competence. If a presented verb was unsuitable, we asked the focus groups to recommend more suitable verb options. Each focus group aimed at agreeing upon the formulation of 10 candidate items and 12 verbs (two for each level of competence). As a result of the focus groups, the candidate items were usually formulated shorter and more succinctly.

As outlined in the methods section, this section aims at describing the evaluation of the developed candidate items in more detail. It is comprised of four phases that are outlined below.

This phase aimed at getting feedback from methodical experts on the designed artefact as a whole. We conducted two expert interviews (Bogner and Menz, 2016; Brinkmann, 2013) with selected methodological experts (professors in the field of education/management and sociology) on the overall structure and concept of the designed artefact. Each lasted about 90 min. We asked the experts to take on the role of a “critical friend”. In the first part of the interview, a member of the research group introduced the expert to the underlying hierarchical taxonomy and the designed artefact. The methodological experts were then asked to comment on the presentation and express what thoughts came to their mind during the introduction.

We asked the experts to comment on the following issues:

1. If any and what “blind spots” the research group had during the development of the artefact

2. Whether the research group made tacit assumptions that were not expressed explicitly but influenced the process

3. Whether there are methodological and scientific risks and opportunities, members of the research group did not express or take into account

4. Further ideas and thoughts from which the development process can profit from

As a result, the project was in total well received by the methodological experts. Suggestions for improvement and feedback were forwarded to the research group and influenced the continuing development process. However, most of the concerns uttered by the experts were also the same as in the focus groups.

This phase was aimed at establishing and evaluating the connection between the hierarchical taxonomy (competence dimensions) and candidate items. While the hierarchical taxonomy was built on a coherent analysis of the O*NET (Fahrenbach et al., 2019a, b), candidate items were developed through a content analysis of qualification standards. In workshops conducted in two schools, we evaluated how well pupils were able to correctly assign a candidate item to the hierarchical taxonomy. We presented the pupils a candidate item and asked them to assign it to one competence dimension and its sub-dimension. The workshops aimed at assigning each candidate item at least 10 times to a dimension of the hierarchical taxonomy.

We conducted two workshops, one at the “Vienna Business School Modling” and the other at “Hohere Technische Lehranstalt Leoben” on 01/07/2019 and 01/16/2019 respectively. A total of 159 pupils aged 14 to 20 (73 female, 86 male) participated in the workshops.

In the first portion of the workshop, we explained the underlying hierarchical taxonomy to the pupils. When possible, we avoided complicated language and students were allowed to ask questions about the taxonomy afterwards. Our aim was that all pupils would understand the hierarchical taxonomy and how they can assign items to a competence dimension and sub-dimension. In the second portion, pupils were asked to assign 15 candidate items to a competence dimension and its respective sub-dimension using a specially programmed application which pupils could access on their smartphones.

We recorded a total of 2,360 assignments of candidate items to a competence and its sub-dimension during the workshops. Each formulation was assigned between 12 and 23 times (median: 16; standard deviation: 2.8). Out of 146 formulations, 76 could apparently be assigned meaningfully by the pupils. For another 70 formulations, the allocation of candidate items by pupils was not meaningful and had to be revised later on (section 5.2.4).

This phase aimed at testing how well pupils were able to interact with the designed artefact (i.e. we set out to test its usability) Usability tests were conducted between 02/05/2019 and 02/15/2019 in three vocational schools in Austria (Landesberufsschule 2, Graz St.-Peter; Landesberufsschule 10, Linz; Caritas Fachschule für wirtschaftliche Berufe Graz and HWL Sozialmanagement Graz) in two federal states in Austria. A total of 188 participants (41 male, 147 female) took part in the test and a subsequent survey (Rubin and Chisnell, 2008). Participants were on average 20 years old (youngest: 17; oldest: 60).

Participants tested the designed artefact in a real-world setting. They had to go through 16 candidate items and conduct a self-assessment of their estimated level of professional competence. In further detail, participants were instructed to read the candidate item carefully and then select the most appropriate verb, indicating their specific level of competence in regard to the item. Afterwards, they were asked to prove their self-assessment with previously predefined documents available in the artefact.

Following this exercise, we handed out a survey to capture the pupils impression. The survey was slightly adapted from (Rubin and Chisnell, 2008). After each session, we implemented the feedback in the artefact. The usability test primarily led to improvements and revisions to the visual design of the artefact. However, pupils could also comment on the candidate items as well as on the overall structure of the artefact.

After carrying out the project phases described above, this phase aimed at a necessary consolidation of the data collected. The hierarchical taxonomy was derived in an abductive procedure (i.e. the competence dimensions proposed in Table 2 should offer an “inference to the best explanation” (Harman, 1965)) from the O*NET. It aims at offering a comprehensive taxonomy to describe professional competences and requirements (Fahrenbach et al., 2019a, b). As the candidate items were derived from five qualification standards, there was a (predictable) imbalance of content between the hierarchical taxonomy and the candidate items. After analysing data from the workshops in vocational schools (assignment of candidate items to competence dimensions in the hierarchical taxonomy), it turns out that the 146 candidate items did not load on all competence dimensions in the hierarchical taxonomy.

Therefore, it was necessary to consolidate the data sets. We consolidated candidate items and the hierarchical taxonomy with the goal of eliminating redundancies in candidate items. Furthermore, we ensured that each competence dimension could be assessed through at least three items. To accomplish this, we eliminated 19, in terms of content redundant, candidate items from the artefact. We developed 31 additional candidate items from the descriptors of the hierarchical taxonomy to represent the missing competence dimensions. These candidate items were–after translation–taken directly from the O*NET and adjusted to style of the other candidate items. To ensure a relatively equal distribution of candidate items within the hierarchical taxonomy, the assignment of candidate items was sharpened based on the results of the focus groups and the school workshops.

Development (see section 5.1) and evaluation (see section 5.2) of candidate items resulted in a hierarchical taxonomy (see Table 2). The hierarchical taxonomy consists of 160 items which can be used to query all transversal competence dimensions it contains. In Table 2, the relatively high number of assignments in the dimensions “Human resources systems and practices” (MC3) and “Business management” (MC9) resulted from the importance indicated in the areas of “personnel administration” and “business administration” in the learning outcomes of the five qualification standards we evaluated. This importance was taken into account by assigning the candidate items to more specific sub-dimensions in the underlying hierarchical taxonomy (MC3.1 - MC3.4 and MC9.1 - MC9.5). The competence dimension “domain knowledge” (DC1) was intentionally not assigned any candidate item, as this dimension is intended as a placeholder for the profession-specific competences per professional association in the ICT artefact.

Practically, we contribute to a conceptual bridge between the American O*NET and the EQF. While we introduced this bridge in (Fahrenbach et al., 2019a, b), we describe the development and evaluation of candidate items for the assessment of transversal professional competences in this paper. We see merit in this bridge because the EQF is far from the best solution possible. During its development, the EQF was subject to intense discussions and negotiations between the parties involved (Guillen et al., 2007). In this sense, the EQF is often deemed “reductionist as it is the lowest common denominator EU member states could agree upon” (Markowitsch and Luomi-Messerer, 2007, p. 50). Contributing this conceptual bridge and showing the compatibility of O*NET and EQF could be a way forward in further developing the EQF.

Besides this more general remark, we outline practical implications for four different stakeholder groups (Achterkamp and Vos, 2007), namely candidates in VPL procedures, assessors, researchers and policy-makers.

First, our research may have implications for candidates in validation procedures. The proposed artefact that can be used as a self-assessment within VPL, especially in the VET sector as it covers a broad range of transversal professional competences. The hierarchical taxonomy and its related assessment items should foster a processes of reflection (Schön, 1983, 1990) in candidates who want to obtain a trading license in the trade and craft sector equivalent to EQF level 6 through the VPL. Furthermore, the hierarchical taxonomy and the assessment items should provide a scaffold that can be used by candidates and be enriched with documentation and proofs of achieved learning outcomes (Council of the European Union, 2012).

Second, our research has implications for assessors who run validation procedures. The hierarchical taxonomy presented in this paper may help to make the assessment of transversal professional competences in VPL procedures less tedious and more efficient. The taxonomy helps assessors to structure documentation provided by candidates. Furthermore, the taxonomy may help assessors who already use other models to reflect upon their own approach and compare their approach to the results presented above. This might help to find blind spots or offer development opportunities.

Third, our research may have implications for other researchers developing taxonomies. On one hand, a hierarchical taxonomy, such as the O*NET content model alone may be insufficient to describe professional competences as it may not be detailed enough and far from situated practice. On the other hand, qualification standards stemming from situated practice may be insufficiently broad to describe professions comprehensively. We aim to offer researchers a way out. The artefact intends to combine the “best of both worlds” - deducting competence dimensions from the O*NET and developing competence statements inductively from qualification standards. As a result, the artefact is able to abductively (Reichertz, 2007) fill the gap between descriptive taxonomies and the lived practice with fitting candidate items from the O*NET. Researchers may use these findings to develop taxonomies which are both, broad and descriptive but situated enough to describe the lived practice.

Fourth, we contribute to the practice of policy-making by showing that the O*NET is far more comprehensive than the five qualification standards derived from the EQF. We show that the O*NET and the EQF are compatible in general and can inform each other. Due to its comprehensiveness, the O*NET may well be able to account for the 80 different jobs in the Austrian trade and craft sector. Policy-makers developing policies regarding the VPL may find this framework valuable. It could be used to develop policies compatible with the American tradition (i.e. the O*NET) and European way to describe qualifications (such as in the EQF). Thus, the hierarchical taxonomy presented could point towards the compatibility of different ways to regulate access to the labour market.

In this paper, we describe the development and evaluation of candidate items within a hierarchical taxonomy that can be used for the assessment of transversal professional competences. In so doing, we build a conceptual bridge between the O*NET and the EQF. In the following, we outline limitations that were mentioned during focus groups and expert interviews, point at further avenues for research and conclude.

Here we outline limitations that were mentioned during focus groups and expert interviews and point at further research endeavours. First, the artefact developed requires that candidates have a good command of the German language as the items are linguistically close to the qualification standards. In this form, the artefact is not designed to validate the prior learning of migrants who do not (yet) possess German skills (Diedrich, 2013, 2017; Souto-Otero and Villalba-Garcia, 2015). Such instruments are already available ^[3,4]. Further research may seek to align the proposed artefact with the European skills, competences, qualifications and occupations ontology, currently developed by the EU ^[5]. Second, the self-assessment of candidates can be subject to cognitive biases, through which candidates consistently overestimate (Forbes, 2005) or underestimate (Beyer, 1990) their level of competences. These biases can be mitigated through triangulation with the documents provided. Further research should clarify inasmuch as people are able to self-assess their professional competence using the “triggerverbs” introduced above and how these verbs each describe a very contextual level of professional knowledge, skills and competences (Bloom et al., 1956; Krathwohl, 2002).

Third, taking into account different definitions and approaches of professional competences (Le Deist and Winterton, 2005; Sandberg and Pinnington, 2009), each definition and approach affords its own methods of assessing. Defining professional competence as applied scientific knowledge (Sandberg and Pinnington, 2009) assessment can be done using surveys and pen-and-paper assessments. Defining professional competence as mainly behavioural (Boyatzis, 1982), a behavioural assessment, such as in simulations is obligatory. Taking into account the European way of defining professional competences more holistically in terms of knowledge, skills, autonomy and responsibility (Council of the European Union, 2017), the assessment should take into account a self-assessment, peer-assessment as well as the assessment of a professional association. Thus, a combination of assessment methods is seen as the gold standard (Cedefop, 2017, p. 74).

Fourth, the design science research paradigm itself is subject to limitations. In order to generate valid results, it requires the rigorous evaluation of the designed artefact. However, a rigorous evaluation and comparison of the artefact is only possible in use. In this regard, the evaluation of the artefact introduced in this paper may be seen as preliminary and not conclusive as it is not compared to other artefacts. Further research may determine the best combination of methods to assess professional competences within an European context.

In sum, we hope that this work proofs to stimulate further research and is of value for practitioners that aim to assess transversal professional competences within the VPL.

The work was partially funded by the EU H2020 program under MSCA-RISE agreement 645751 (RISE BPM) and FFG Austrian Research Promotion Agency (project number: 866270). I am very grateful to anonymous reviewers that helped to improve the paper.