Table A2 Coding scheme for evaluation...

Table A2

Coding scheme for evaluation interviews

ID	Code	Example(s)	Short description
1) General comments
1a	Labels	“another thing that you call this performance dimension, and then you have here process performance, and that's super confusing. Which performance we're talking about?”	Issues with understanding the names of some elements in the prototype
1a	Labels	“I found that a little bit the naming of things wasn't very intuitive to me.”
1b	Interface	“This is very clear, this red and green colours are used sparsely. And it's good. It's good, because too much colours are horrible when you try to analyze something.”	Comments related to the interface
1b	Interface	“I think the visualisation is perfect, because it's clear, it's concise, it's simple.”	Comments related to the interface
1c	Suggestions	“So you can see all KPIs compared. And you can, instead of like - yeah, I see that it’s not clickable now - but instead of clicking, selecting, if you just have three, four KPIs, just show them.”	Direct suggestions from interviewees on what to change in the visualisation
1c	Suggestions	“In fact, I want to be able to put a custom dimension, ”This is what time means to me in this process. This is what matters.”
2) Improvement opportunity identification
2a	Understanding current problem	“But before I go there, I just want to familiarize myself with the actual process and some of the parameters.”	Voicing the need to see the problem that is dealt with first before going to evaluate the redesign options
		“but before I would like to see the redesign opportunities, I would kind of go through the identified issues here.”
		“if this redesign option does not address that problem, then it doesn't help.”
2b	Understanding context data	“Well, it would be nice if I could see the data underlying that. You know, when I click the arrow, for example here, that I can see the specific cases, perhaps a list of specific cases that are affected by this potential improvement.”	Voicing the need to understand what the suggestions in the visualisation of improvement opportunities are based on
2b	Understanding context data	“We can maybe have the opportunity to put a click on the flow and see some data, some details that let you understand deeply the work of the secretary alone.”
3) Parameters for prioritisation
3a	Change in KPIs	“So if I can see that there is something that can reduce the rework - great, I would look at that.”	Evaluating the change in KPIs (process performance) to prioritise improvement opportunities
		“for deciding on the specific improvement opportunity, the important thing for me to understand is how they change the process performance”
		“I would focus on changes in KPIs. So I would try to look what exactly changes? And if it's significant?”
3b	Feasibility	“That's the one thing, the difference between the output or the result of improvements, but also I'm thinking of some effort that I have to put to resolve this issue.”	Analyzing the trade-off between the gains and the feasibility of implementation (cost, effort)
		“First, how many cases are affected? So when it's just 5%, maybe I will not do anything. So this is important. So when I go to overview, I will look at this. And then costs that I would pay for improvement for this suggestion for this opportunity.”
		“I will try to see if I can, how do I make the biggest improvement without devoting extra resources”
3c	Combined impact of improvement opportunities	“we'll try to see how the different combinations between these opportunities work”	Assessing if it's possible to address several improvement opportunities at once
3c	Combined impact of improvement opportunities	“If I can by solving the problem two, probably I can already solve half of problem number one. So we always have to consider the solutions, if they could actually be solved concurrently, at the same time”
3d	Consulting context data	“So rather than just show me like we saw here, just a quick comparison - this is definitely nice - but the next question is, okay, why do you think it's going to go by 10%? Show me the underlying data.”	Consulting with the context data on which the calculations for changes were made before making the decisions
3d	Consulting context data	“I would want more clarity, how things, what do they mean, what exactly is meant by a certain KPI, or how did certain suggestion come about?”
4) Communication
4a	Things communicated	“Oh, certainly use images from the application and therefore, it's to support my decision by data.”	In which form interviewees would present the findings from the prototype
4a	Things communicated	“but what's important that you can just screenshot the graphs from here.”
4b	Methods for communication	“So in this case, I would, I would have just a few slides, maybe like one or two slides showing the current process and the improvement proposal or the improvement opportunity identified by the software, and the alternative that I like the most or the alternative that I would like to sell.”	What methods interviewees would use to present the findings from the prototype
4b	Methods for communication	“The best way, it's not a PowerPoint presentation but directly in the tool, directly this data. Fact based, it's top one.”

ID	Code	Example(s)	Short description
1) General comments
1a	Labels	“another thing that you call this performance dimension, and then you have here process performance, and that's super confusing. Which performance we're talking about?”	Issues with understanding the names of some elements in the prototype
1a	Labels	“I found that a little bit the naming of things wasn't very intuitive to me.”
1b	Interface	“This is very clear, this red and green colours are used sparsely. And it's good. It's good, because too much colours are horrible when you try to analyze something.”	Comments related to the interface
1b	Interface	“I think the visualisation is perfect, because it's clear, it's concise, it's simple.”	Comments related to the interface
1c	Suggestions	“So you can see all KPIs compared. And you can, instead of like - yeah, I see that it’s not clickable now - but instead of clicking, selecting, if you just have three, four KPIs, just show them.”	Direct suggestions from interviewees on what to change in the visualisation
1c	Suggestions	“In fact, I want to be able to put a custom dimension, ”This is what time means to me in this process. This is what matters.”
2) Improvement opportunity identification
2a	Understanding current problem	“But before I go there, I just want to familiarize myself with the actual process and some of the parameters.”	Voicing the need to see the problem that is dealt with first before going to evaluate the redesign options
		“but before I would like to see the redesign opportunities, I would kind of go through the identified issues here.”
		“if this redesign option does not address that problem, then it doesn't help.”
2b	Understanding context data	“Well, it would be nice if I could see the data underlying that. You know, when I click the arrow, for example here, that I can see the specific cases, perhaps a list of specific cases that are affected by this potential improvement.”	Voicing the need to understand what the suggestions in the visualisation of improvement opportunities are based on
2b	Understanding context data	“We can maybe have the opportunity to put a click on the flow and see some data, some details that let you understand deeply the work of the secretary alone.”
3) Parameters for prioritisation
3a	Change in KPIs	“So if I can see that there is something that can reduce the rework - great, I would look at that.”	Evaluating the change in KPIs (process performance) to prioritise improvement opportunities
		“for deciding on the specific improvement opportunity, the important thing for me to understand is how they change the process performance”
		“I would focus on changes in KPIs. So I would try to look what exactly changes? And if it's significant?”
3b	Feasibility	“That's the one thing, the difference between the output or the result of improvements, but also I'm thinking of some effort that I have to put to resolve this issue.”	Analyzing the trade-off between the gains and the feasibility of implementation (cost, effort)
		“First, how many cases are affected? So when it's just 5%, maybe I will not do anything. So this is important. So when I go to overview, I will look at this. And then costs that I would pay for improvement for this suggestion for this opportunity.”
		“I will try to see if I can, how do I make the biggest improvement without devoting extra resources”
3c	Combined impact of improvement opportunities	“we'll try to see how the different combinations between these opportunities work”	Assessing if it's possible to address several improvement opportunities at once
3c	Combined impact of improvement opportunities	“If I can by solving the problem two, probably I can already solve half of problem number one. So we always have to consider the solutions, if they could actually be solved concurrently, at the same time”
3d	Consulting context data	“So rather than just show me like we saw here, just a quick comparison - this is definitely nice - but the next question is, okay, why do you think it's going to go by 10%? Show me the underlying data.”	Consulting with the context data on which the calculations for changes were made before making the decisions
3d	Consulting context data	“I would want more clarity, how things, what do they mean, what exactly is meant by a certain KPI, or how did certain suggestion come about?”
4) Communication
4a	Things communicated	“Oh, certainly use images from the application and therefore, it's to support my decision by data.”	In which form interviewees would present the findings from the prototype
4a	Things communicated	“but what's important that you can just screenshot the graphs from here.”
4b	Methods for communication	“So in this case, I would, I would have just a few slides, maybe like one or two slides showing the current process and the improvement proposal or the improvement opportunity identified by the software, and the alternative that I like the most or the alternative that I would like to sell.”	What methods interviewees would use to present the findings from the prototype
4b	Methods for communication	“The best way, it's not a PowerPoint presentation but directly in the tool, directly this data. Fact based, it's top one.”

[ViewLarge]

Sharing Unavailable