Table 1 Summary of recommendations to... | Emerald Publishing

Table 1

Summary of recommendations to CHAPE

#	Requirements	Recommendations
1	Human agency and oversight	Consider how a false diagnosis could affect the end user's decision making Keep humans in the loop to prevent overconfidence in or overreliance on the AI system Ensure there are opportunities for human intervention and verification in the system's decision process
2	Technical robustness and safety	To consider extreme cases or unusual environments Monitoring increased prescriptions and other anomalous decisions Log of procedures and decisions made Try to approach accuracy achieved by junior medical professionals Consider the (human) cost of an incorrect medical recommendation
3	Privacy and data governance	If including sensitive/proprietary patient data does not significantly improve the effectiveness of the system, it should not be used Any of this confidential data that is used should be encrypted and anonymized Keep detailed logs of data sourcing and access/use, as well as citing info in the recommendations given by the model Provide the relevant data quality, governance and cybersecurity training for developers, data scientists, medical personnel and other service evaluators
4	Transparency	The importance of any variables used to inform the system's output should be communicated clearly The purpose of the system, people empowerment to enhance the medical skills of the general populace, should be clearly stated Provide clear visualizations of how the strength of each user input (personal info, medical history, current symptoms) contribute to the decision
5	Diversity, non-discrimination, and fairness	Any dataset used to train a machine learning involved must undergo thorough study to ensure it is unbiased Consider some limitations to the datasets, (e.g. patients 18+, historical data), or make them available them only for specific organizations Considerations should be made at the design phase for people with disabilities, special needs or who are at the risk of exclusion, e.g. colourblind friendly palettes, adjustable text size and font, text to speech service Develop a mechanism to incorporate the involvement end users, doctors, AI experts/development
6	Societal and environmental wellbeing	Ensure probabilities/confidence in model output is communicated. If it is unsure of any diagnosis, request more info from the user A/B testing on critical aspects of the UI to give rich feedback on the clarity of communicating suggestions made by the system Provision should be made to sustainably source the materials and energy required to store the data and host the system on a server The deployment of an AI solution is envisaged have a lower environmental impact than the human labour equivalent
7	Accountability	Design an iterative process whereby trained evaluators (representative sample of stakeholders) provide feedback on model recommendations Consider the trade-offs to be made: ○ Model performance vs data privacy ○ Disease severity and available medical resources Routine risk assessments for o end user's misdiagnosis ○ Medical professionals-confirmation bias (over reliance on the system) ○ Developers of the AI system lack of robustness/stress testing of model

#	Requirements	Recommendations
1	Human agency and oversight	Consider how a false diagnosis could affect the end user's decision making Keep humans in the loop to prevent overconfidence in or overreliance on the AI system Ensure there are opportunities for human intervention and verification in the system's decision process
2	Technical robustness and safety	To consider extreme cases or unusual environments Monitoring increased prescriptions and other anomalous decisions Log of procedures and decisions made Try to approach accuracy achieved by junior medical professionals Consider the (human) cost of an incorrect medical recommendation
3	Privacy and data governance	If including sensitive/proprietary patient data does not significantly improve the effectiveness of the system, it should not be used Any of this confidential data that is used should be encrypted and anonymized Keep detailed logs of data sourcing and access/use, as well as citing info in the recommendations given by the model Provide the relevant data quality, governance and cybersecurity training for developers, data scientists, medical personnel and other service evaluators
4	Transparency	The importance of any variables used to inform the system's output should be communicated clearly The purpose of the system, people empowerment to enhance the medical skills of the general populace, should be clearly stated Provide clear visualizations of how the strength of each user input (personal info, medical history, current symptoms) contribute to the decision
5	Diversity, non-discrimination, and fairness	Any dataset used to train a machine learning involved must undergo thorough study to ensure it is unbiased Consider some limitations to the datasets, (e.g. patients 18+, historical data), or make them available them only for specific organizations Considerations should be made at the design phase for people with disabilities, special needs or who are at the risk of exclusion, e.g. colourblind friendly palettes, adjustable text size and font, text to speech service Develop a mechanism to incorporate the involvement end users, doctors, AI experts/development
6	Societal and environmental wellbeing	Ensure probabilities/confidence in model output is communicated. If it is unsure of any diagnosis, request more info from the user A/B testing on critical aspects of the UI to give rich feedback on the clarity of communicating suggestions made by the system Provision should be made to sustainably source the materials and energy required to store the data and host the system on a server The deployment of an AI solution is envisaged have a lower environmental impact than the human labour equivalent
7	Accountability	Design an iterative process whereby trained evaluators (representative sample of stakeholders) provide feedback on model recommendations Consider the trade-offs to be made: ○ Model performance vs data privacy ○ Disease severity and available medical resources Routine risk assessments for o end user's misdiagnosis ○ Medical professionals-confirmation bias (over reliance on the system) ○ Developers of the AI system lack of robustness/stress testing of model

Source(s): Table created by author