Table 1 Overview of the procedures... | Emerald Publishing

Table 1

Overview of the procedures comprising the development and evaluation of candidate items for the self-assessment

Design science phase	Goal	Input	Activity	Output
Development (Fahrenbach et al., 2019a)	Development of a comprehensive hierarchical taxonomy of professional competences	Occupational information network (Peterson et al., 2001) and European competence taxonomies (Le Deist and Winterton, 2005; Mulder et al., 2007)	Development of a hierarchical taxonomy consisting of 32 sub-competences through haptic clustering and abductive reasoning	Hierarchical taxonomy with four competence dimensions and 32 sub-competence dimensions (Fahrenbach et al., 2019a)
Development (section 5.1.1)	Development of candidate items to describe the competence dimensions in the hierarchical taxonomy	Qualification standards of five occupations from the Austrian trade and craft sector	Qualitative content analysis of five qualification standards (Krippendorff, 2009; Mayring, 2015) and iterative process with several feedback loops from Austrian federal economic chamber	Set of 146 candidate items to describe transversal professional competences
Development (section 5.1.2)	Identification of critical stakeholder groups to iteratively revise candidate items	Workshop with members of the Austrian federal economic chamber	Identification of critical stakeholders according to Achterkamp and Vos (2007)	Five critical stakeholder groups
Development (section 5.1.3)	Revise 146 candidate items and establish intersubjective understandability	146 candidate items and proposed verbs for each competence level	Establishment of intersubjective understandability of candidate items through focus groups with representatives of five critical stakeholders groups	Intersubjective understandable set of 146 candidate items with 2–4 fitting verbs per level of competence (novice to expert)
Evaluation (section 5.2.1)	Gather feedback from methodical experts on overall approach	Presentation about the research design, methods and research process	Two expert interviews conducted via Skype, asking for feedback	Feedback was incorporated into the development process
Evaluation (section 5.2.2)	Assignment of intersubjective understandable items to hierarchical taxonomy	Set of 146 intersubjective understandable items and hierarchical taxonomy (Fahrenbach et al., 2019b)	Assignment of items to hierarchical taxonomy through workshops in two vocational schools with 159 pupils	76 items could be meaningfully assigned by pupils. 70 items could not be meaningfully assigned to the hierarchical taxonomy
Evaluation (section 5.2.3)	Test the perceived usability of the programmed ICT artefact	Programmed ICTs-artefact with incorporated items	Usability testing with surveys in three vocational schools with 188 participants. Gathering feedback	Feedback was incorporated into the next design iteration of the ICT artefact
Evaluation (section 5.2.4)	Consolidation of the data acquired in previous phases of the process	Need for data consolidation and a coherent set of items fitting to hierarchical taxonomy	Consolidation through development and inclusion of new items (31) from the hierarchical taxonomy and eliminating redundancies (19 items)	Hierarchical taxonomy with 4 competence dimension, 32 sub-competence dimensions, 160 items with established intersubjective understandability and 2 to 4 fitting verbs per level and item to assess the level of professional competence

Design science phase	Goal	Input	Activity	Output
Development (Fahrenbach et al., 2019a)	Development of a comprehensive hierarchical taxonomy of professional competences	Occupational information network (Peterson et al., 2001) and European competence taxonomies (Le Deist and Winterton, 2005; Mulder et al., 2007)	Development of a hierarchical taxonomy consisting of 32 sub-competences through haptic clustering and abductive reasoning	Hierarchical taxonomy with four competence dimensions and 32 sub-competence dimensions (Fahrenbach et al., 2019a)
Development (section 5.1.1)	Development of candidate items to describe the competence dimensions in the hierarchical taxonomy	Qualification standards of five occupations from the Austrian trade and craft sector	Qualitative content analysis of five qualification standards (Krippendorff, 2009; Mayring, 2015) and iterative process with several feedback loops from Austrian federal economic chamber	Set of 146 candidate items to describe transversal professional competences
Development (section 5.1.2)	Identification of critical stakeholder groups to iteratively revise candidate items	Workshop with members of the Austrian federal economic chamber	Identification of critical stakeholders according to Achterkamp and Vos (2007)	Five critical stakeholder groups
Development (section 5.1.3)	Revise 146 candidate items and establish intersubjective understandability	146 candidate items and proposed verbs for each competence level	Establishment of intersubjective understandability of candidate items through focus groups with representatives of five critical stakeholders groups	Intersubjective understandable set of 146 candidate items with 2–4 fitting verbs per level of competence (novice to expert)
Evaluation (section 5.2.1)	Gather feedback from methodical experts on overall approach	Presentation about the research design, methods and research process	Two expert interviews conducted via Skype, asking for feedback	Feedback was incorporated into the development process
Evaluation (section 5.2.2)	Assignment of intersubjective understandable items to hierarchical taxonomy	Set of 146 intersubjective understandable items and hierarchical taxonomy (Fahrenbach et al., 2019b)	Assignment of items to hierarchical taxonomy through workshops in two vocational schools with 159 pupils	76 items could be meaningfully assigned by pupils. 70 items could not be meaningfully assigned to the hierarchical taxonomy
Evaluation (section 5.2.3)	Test the perceived usability of the programmed ICT artefact	Programmed ICTs-artefact with incorporated items	Usability testing with surveys in three vocational schools with 188 participants. Gathering feedback	Feedback was incorporated into the next design iteration of the ICT artefact
Evaluation (section 5.2.4)	Consolidation of the data acquired in previous phases of the process	Need for data consolidation and a coherent set of items fitting to hierarchical taxonomy	Consolidation through development and inclusion of new items (31) from the hierarchical taxonomy and eliminating redundancies (19 items)	Hierarchical taxonomy with 4 competence dimension, 32 sub-competence dimensions, 160 items with established intersubjective understandability and 2 to 4 fitting verbs per level and item to assess the level of professional competence