Chapter 9: eHealth and Telemedicine for Risk Prediction and Monitoring in Kidney Transplantation Recipients:
-
Published:2024
Wiebke Duettmann, Roland Roller, Bilgin Osmanodja, Verena Graf, Matthias Pfefferkorn, Danilo Schmidt, Anne Schwerk, Marcel G. Naik, Klemens Budde, 2024. "eHealth and Telemedicine for Risk Prediction and Monitoring in Kidney Transplantation Recipients: ", Technology in Healthcare: Introduction, Clinical Impacts, Workflow Improvement, Structuring and Assessment, Brian Pickering, Roland Roller, Holmer Hemsen, Gerrit J. Noordergraaf, Igor Paulussen, Alyssa Venema
Download citation file:
9.1 Introduction
Patients with end-stage chronic kidney disease need dialysis or kidney transplantation for survival. Many trials confirm that transplantation is the favored option for many patients because of better survival, better quality of life, and lower costs [1–6]. However, many Kidney Transplant Recipients (KTRs) suffer from multiple complications such as rejection, infection, underlying diseases, and side effects of their medication [7,8]. To detect complications or even avoid them through early interventions, patients must comply with the strict aftercare program, including regular visits to the transplant center, the timely intake of immunosuppressants, the regular measurement of vital signs (to control cardiovascular risks), and a special sensitivity for changes in their health status [9]. Hence, the contributions of the patient, adherence, and patient empowerment are key to successful long-term outcomes [10].
However, some studies showed low adherence to instructions by KTRs and that non-adherence frequently contributes to graft loss [11]. While most KTRs tightly follow the aftercare schedule directly after transplantation, adherence progressively drops over the years. Interestingly, adherence in the first month after transplantation has been shown to be predictive of the long-term organ outcome [11]. Humans can be distinguished into various adherence types with typical patterns, which can be supported by different methods [12,13]. Therefore, it is important to identify the fewer adherent patients and support them with advice and education and provide a structure which enables these patients to follow their instructions [11]. This process is called empowerment and is crucial for adherence. This empowerment, education, and support must be lifelong. Consequently, adherent or partially adherent patients should also be rewarded with constant support, praise, and sympathy for further motivation.
eHealth products such as mobile phone applications (apps) may have the potential to improve adherence in chronically ill patients and in acute or special medical circumstances such as pregnancy. Apps are permanently available on smartphones and thereby may provide constant support or support upon demand. However, such mobile phone apps improve adherence only if the patients consider the app useful [14]. Ideally, the support system includes some additional human support, typically from friends and family members by healthcare professionals [6].
To integrate this knowledge, we designed a telemedicine service with a mobile phone app, where a telemedicine team in the transplant center supports the KTRs to better empower the patient and increase patient adherence. The integration of this support system into the workflow to ease the working processes of the medical staff was another key aspect. In close interaction with patients, nurses, medical doctors, and computer scientists, we developed this telemedical module, including an app for patients, interfaces, and a front-end for telemedicine staff (a dashboard). The key component of the telemedicine concept was to empower patients to better take care of their health and improve medical and therapeutic adherence. Better adherence should lead to a lower complication rate, e.g., better blood pressure control and a lower frequency of Donor-Specific Antibodies (DSA), which may cause rejections and are an important cause of graft loss. The timely detection of complications such as infections, rejections, or side effects of immunosuppressants is crucial for a better response to treatment and the avoidance of serious, life-threatening clinical courses.
The main goal of our study was to detect critical patients early in order to reduce complications and thereby reduce hospitalizations, which will reduce costs and improve the quality of life of patients. To select and detect critical patients, a telemedicine team was established to regularly evaluate incoming data and take medical action if needed. To support the telemedicine team, which may be overwhelmed with large numbers of constantly incoming data, we developed risk prediction models to better forecast critical endpoints such as rejection, graft loss, and infection [15].
9.2 Challenges
To set up a telemedicine system in the nephrology department at Charité, various challenges had to be addressed, including organizational issues, hospital bureaucracy, and, most importantly, data protection. The Charité has its IT hierarchy and working structure highly protected with a firewall. Thus, applications for the use of specific plug-ins for accessing the internet and the usage of external hardware were necessary. Bureaucracy about different application forms was time-consuming. External hardware might contain computer viruses or spyware; therefore, it had to be evaluated before connecting to the network. In addition, an extensive evaluation of data protection regulations was required. The hospital and each federal state regulate data protection aspects and are different as well as strict. An external lawyer, who is specialized in patient rights and General Data Protection and Regulations (GDPR), evaluated the data protection concept of the telemedicine module. In cases of cooperation with other institutes, guest science contracts must be confirmed. The organizational aspects included a discussion of the necessity and safety of such a new telemedicine project and the need for personnel, equipment, and rooms. For a sustainable solution, negotiations with health insurance companies were necessary, and finally, a contract for financial support was signed and discussions on the potential use of telemedicine services in the future were formalized.
From a technical point of view, no blueprint or standard solution for a telemedicine approach existed. Particularly, integration into the existing hospital infrastructure was not trivial. Because good integration of any telemedicine solution into the workflow is essential for success, we aimed to integrate the dashboard into our patient documentation system, TBase [16]. A telemedicine module has fundamentally new requirements on safety and data protection such that certain safety and privacy aspects had to be evaluated again (privacy impact assessment, etc.).
Unlike the telemedicine system, automatic risk prediction was planned as a proof-of-concept analysis. Therefore, it was not meant to be integrated directly into the telemedicine system or used in direct clinical care for patients since this would require a preemptive and thorough risk-benefit assessment according to the new medical device regulations. The application of risk prediction in a research setting still induces many challenges from the legal side. From a technical point of view, various other problems also had to be addressed. This included access for external researchers, as data had to be processed on Charité premises with limited hardware resources to train machine-learning models. Moreover, as (real-world) clinical data are noisy, they require a great deal of expert and clinical knowledge for adequate processing. For instance, data fields in the database changed over time. In addition, working on data in a closed “ecosystem” (which cannot be shared) reduces the possibility of comparability (of machine learning models). In this way, it is difficult to learn from existing work, generate similar cohorts, and reproduce systems and results. Other challenges were the interdisciplinary aspects and multilingualism of the teams.
Finally, personnel for a telemedicine team had to be recruited without an existing job description for this new field of medicine. We, therefore, decided to look for experienced healthcare workers with an expressed interest in eHealth. The next step was to create a new working process for telemedicine aftercare with standard operating procedures and to integrate this process into the routine transplant aftercare in outpatient clinics without interfering with the current medical staff [9]. Finally, yet importantly, patients had to be convinced, and after signing a newly developed informed consent form, the formal onboarding process had to be started. Not surprisingly, many time-consuming issues around the download and installation of the app on the patient’s mobile phone had to be resolved.
9.3 Methods
The study addresses the following two goals:
Establishing a telemedicine system
Development of a risk prediction system
9.3.1 Telemedicine
The telemedicine system includes a telemedicine team, which evaluates incoming data from patients on a daily basis and can react to problematic data with advice and support. The digital infrastructure of the telemedicine system includes a patient app and a telemedicine dashboard. The dashboard is integrated within an approved Electronic Medical Record (EMR) called TBase, which is used for routine documentation of patients with kidney diseases at Charité. TBase is also a research database, which allows scientists to work on the data without having access to the personal data of the patients [16]. The patient app was designed by an interdisciplinary team consisting of nurses, patients, medical doctors, and computer scientists and implemented by an industry partner. The app sends data to TBase via an interoperable Health Level 7 (HL7) Fast Healthcare Interoperability Resources (FHIR) Application Programming Interface (API) [9].
A free (of charge) mobile phone app enables patients to document their medication intake, vital signs (weight, blood pressure, pulse, temperature, blood sugar value, and blood oxygen saturation), as well as their sense of well-being. Moreover, the app includes a medication alert and direct contact options for the telemedicine team, which include chat functionality and potentially a video option. Our telemedicine team is available on working days between 7 am and 4 pm. In addition, the app allows for the exchange of documents, laboratory values, and medication plans (Figure 9.1). This means the patient is enabled to have all relevant information ‘in his/her hands’ (‘empowerment’) and has access to an easy personal exchange with the telemedicine team (‘support’).
Examples of patient app.
An FHIR server distributes the data between the patient app, TBase at the transplant center, and other users (such as the data system of home nephrologists). A telemedicine dashboard within TBase displays the data in the transplant center for the physician and telemedicine team (Figure 9.2).
The dashboard was also developed in cooperation between nurses, medical doctors, and computer scientists. The dashboard is constantly adapted to the needs of the telemedicine team and patients. An underlying alert system allows an easy evaluation of adherence (e.g., how often do patients share vital signs with the telemedicine team?). Based on the standard operating procedures and the incoming data, the telemedicine team decides when and how to contact the patient or the home nephrologist. The standard operating procedures contain a range of critical vital signs that will cause an immediate or intermediate reaction (Figure 9.3). For instance, if a patient shows raised blood pressure values for a month, the patient is flagged, and the telemedicine team potentially intervenes with different measures, such as intensified antihypertensive therapy. If patients are not adherent to the project (sending no data) and the dialogue with the patient confirms non-adherence (sending no values and do not measure vital signs), the telemedicine team aims to find out the individual resistance to adherence and to help the patient improve their adherence.
TBase dashboard.
Vital signs and medical interpretation with suggestions for actions.
A key component of the telemedicine concept is strict compliance with GDPR. As mentioned above, we first designed the concept, let the concept be evaluated by an independent specialized law firm and the data protection units, and then started with the implementation. Key elements are the firewall at Charité, the FHIR authentication server system, the HL7 FHIR APIs, and the data encryption system. TBase is protected by the Charité firewall. The HL7 FHIR API receives and distributes data from external sources to internal sources and vice versa. Patients have a TBase ID, a telemedicine ID, and the system ID of the home practitioner. The telemedicine module has an FHIR server that receives incoming data, encrypts and decrypts them, and distributes them. This server is located externally on a protected server within Germany. The key to the encryption code lies with the third party that developed the app. The digital infrastructure in which the TBase is embedded is presented in Figure 9.4.
Overview of the architecture of TBase within Charité.
To include a patient in the system, a sophisticated multi-step onboarding system is necessary. First, the medical doctor verifies the patient’s identity and the inclusion documents. Then the telemedicine team activates the onboarding process in TBase. This causes an FHIR request to the FHIR server for an empty onboarding sheet. TBase fills in the patient ID and sends it back with self-chosen login data. Now, the patient can open the app with the login data. The patient must change the login data so that these are individualized and private. If a general practitioner is to be included for a specific patient, the same process can be used.
9.3.2 Risk Prediction Models
The risk prediction model targeted the identification of patients at risk of infection, rejection, and/or graft failure. The aim is to identify patients before a critical situation appears, thus preventing that critical event as well as offering support to a physician with appropriate information. This, in turn, should reduce hospitalizations, the duration of the stay, and the intensity of diagnostics and therapy. Fewer hospitalizations will reduce costs and improve the patient’s quality of life. The risk prediction model is based on de-identified (removal of names, addresses, etc.) data in TBase. The endpoints of interest were developed in close collaboration between computer scientists and physicians, which are the detection of transplant loss, rejection, and infections within the next 90 days. After extensive data cleansing steps to deal with missing, noisy, and false information, different cohorts (data subsets) were generated.
The primary data source of our risk prediction models is TBase, an EHR system containing highly granular KTR data spanning the last 20 years from different sources, such as the hospital system as well as pathology, laboratories, and other systems. TBase was designed in the 1990s, when prospective data collection of all available data from KTRs started as the main documentation system of the Berlin Kidney Transplant Centre. Access to the data is granted to the scientific and medical staff of the Medical Department of Nephrology and Medical Intensive Care, Berlin, Germany. It should be pointed out that all partners in this study underwrote the European Regulation 2016/679. TBase includes structured and unstructured data and is stored in a relational database. In 2019, TBase was fundamentally reengineered, and a general telemedicine dashboard was introduced.
As data could not leave the hospital, the model development was conducted within the given infrastructure: first physically within the hospital and then later, during the coronavirus disease 2019 (COVID-19) pandemic, using a Virtual Private Network (VPN) access. The model was trained on a Linux cluster, which had been purchased for this project and was located in a secure space inside the hospital.
We relied on Python and the scikit-learn library for data processing and model development. Instead of starting with a complex model, we decided to start with a simple and robust baseline system that can integrate suggestions by the physicians (expert knowledge). More precisely, we relied on Gradient Boosted Regression Trees (GBRT). To train our model, we use retrospective data from TBase from Charité Mitte and Virchow Klinikum from 2008 to 2020. The data includes more than 4,500 patients with several million Data Points (DPs). A DP describes a moment in a patient’s life, or more specifically, a moment when new data about a patient is inserted into TBase. Each of those DPs is used to make new predictions.
Following the suggestions of the physicians, various patients’ DPs have been filtered out beforehand to exclude:
DPs of patients below 18 (at that point in time),
DPs with target endpoint currently not active,
DPs within a time frame of fewer than a week after an infection or a rejection,
DPs within a time frame of fewer than 2 weeks after a transplant, and
DPs of patients who do not have a follow-up DP in the next 15–180 days.
The model integrates semi-structured data, including:
Socio-demographics (e.g., gender, age, smoking status)
Vitals (e.g., blood pressure, weight)
Primary disease
Medications
Laboratory values
Transplant/donor information
Previous diagnoses
Admissions to the hospital
Information comes at different intervals, is updated infrequently at different times depending on the current situation change of the patient, and comes in different formats. Some data, like socio-demographics or transplant information, are relatively static. The models consist of 300 trees and integrate about 300 features using about 100,000 DPs from about 1,400 patients. However, during its development, it turned out that our GBRT ‘baseline’ already provided promising results (see next paragraph). Thus, instead of integrating a more complex model, we wanted to find out how well physicians can solve the task and if the current system is good enough to support physicians in their daily routine.
9.4 Results
9.4.1 Telemedicine
After signing a contract (according to § 140 German Social Law) with two major health insurance companies in Germany and after the implementation of the telemedicine unit, the telemedicine team started to sign up KTRs in February 2020. Since then, more than 450 KTRs have been treated up to October 2022. The overall acceptance rate was high; only a few dropouts occurred, and patient satisfaction was high. Since the beginning, 95 KTRs dropped out (8.53%), and 38 (3.2%) were excluded by the telemedicine team due to persistent non-participation despite contacts with the team. Other reasons for dropping out or exclusion were technical issues with the app, a lack of satisfaction with the app (usability), and a lack of interest in the project.
The average age of participants is 52 years, and 63.4% of patients are male. The KTRs are mostly transplanted for the first time (85.5%; N = 384). Only 104 KTRs (12.12%) received a second transplant, and 20 KTRs (1.43%) even received a third one. In 39 cases (3.93%), combined transplantation of the kidney and pancreas occurred. Initially, the transplant had been in situ for six years before inclusion in the project. Later, the telemedicine team included almost all newly transplanted patients.
Frequency of incoming data distributed to the months since transplantation. X-axis: Time points of measurement in months. Y-axis: Numbers of measurements per month. Legend: Duration of transplantation until inclusion into the project <1.5 months, >12 months, 1.5–6 years, and 6–12 years.
Out of 84,130 incoming blood pressure data, 1,285 were critical (1.5%; Figure 9.3) and led to a contact by a member of the telemedicine team. The same was true for 84,127 heart frequency data, of which 3.337 (4%) were critical (Figure 9.3). Out of 68,564 temperature data, 458 (0.7%) were critical (Figure 9.3). Moreover, of 14.631 blood sugar values, 104 (0.7%) were critical (60– 140 mg/dl or 3.3–7.8 mmol/l) and led to contact by the telemedicine team. The incoming data distribution since the start of the project is shown in Figure 9.5. Patients used the telephone, email, and chat to get in touch with the telemedicine team during working days and between 7 am and 4 pm. These frequencies have not been evaluated yet.
The other pillar of telemedicine care is acute consultation in cases of symptoms, prescriptions, appointment service, laboratory tests, and other medical or psychosocial problems. During the COVID-19 pandemic, a major task was COVID-19 counselling in case of positive COVID-19 polymerase chain reaction (COVID19-PCR) swabs, treatment, and monitoring of KTRs with mild and moderate COVID-19 infection.
To round up the telemedicine care, nine nephrologists in private practice were included in the system for data exchange (laboratory values, medical data, vital signs, and soon the progress note of the attending doctor).
9.4.2 Risk Prediction Models
The risk prediction has been evaluated in two different setups: first, as a retrospective internal study and evaluated within cross-validation (70% training, 15% development, 15% test), and second, within a small reader study, the performance of the machine-learning model was compared to that of physicians with and without automatic decision support.
The first results from the retrospective data predicting the three different endpoints for the next 90 days (endpoints: rejection, graft loss, or infection) were overall promising in terms of high Area Under the Curve (AUC) and Receiver Operating Characteristic (ROC) scores (between 80 and 95 regarding the different aims). However, as medical data can typically not leave the hospital, there was no comparison to other, similar approaches. Feature engineering partially led to minor system improvements.
For the reader study, an ad hoc convenience sample of 120 patients was selected. The random selection was conducted so that we could ensure that a minimum of 20 patients would be within the risk scope of each endpoint. Eight medical doctors in different stages of education and years of work experience (four junior physicians and four senior physicians) were recruited to participate. The reader study was conducted in two parts: first, each physician received 15 patients to examine and had to estimate the likelihood that (at least one of) our endpoints would occur within the next 90 days. Each physician could take up to 30 minutes to examine the medical history of each patient. In the second part, each physician was assigned to 15 different patients, and this time they also received the risk estimation of our Clinical Decision Support System (CDSS; Figure 9.6.) in order to reach their decision.
The study showed that (1) the risk prediction system tends to outperform medical doctors according to AUC-ROC, (2) senior medical doctors did not necessarily improve their AUC-ROC performance when offered the risk prediction system in addition to their own analysis of the data, only junior physicians did, (and 3) medical doctors and risk prediction partially found different patients at risk. More details about the models, experiments, results, and learnings can be found in [15,17].
This dashboard has been used in the second part of the study. It shows the current risk for a given endpoint (see red arrow) as well as the previous risk scores over time, mapped to a kind of traffic light system. The color red for instance indicates that there is a higher risk that the endpoint might occur within the next 90 days. On the right side, relevant local (relevant for this decision) and global (relevant for the task in general) features are presented, which had an influence on the model’s estimation [17].
9.5 Learnings
First of all, if one intends to work with sensitive patient data, the data protection unit should be contacted as early as possible, and the concept should be presented in every detail. The data streams should be sketched and evaluated by data protection experts from the beginning. When the security architecture is approved, interfaces can be activated. The data should be stored in the European Union (EU) and cannot leave it. No services from Non-EU providers can be used, e.g., to monitor health activities via an app. Another important issue is to rely on standardized software (e.g., HL7 FHIR standard, FHIR server, Systematized Nomenclature of Medicine – Clinical Terms (SNOMED CT), Logical Observation Identifier Names and Codes (LOINC), Patient Related Outcome Measurements (PROM)).
Another important aspect to consider is that medical data are noisy, incomplete, without context information, and contain unstructured information. If values are missing, for example, it does not automatically mean that the dataset is incomplete and must be deleted for the greater good. Extensive cooperation of data scientists with medical doctors is key for understanding, and time-consuming data cleansing is needed in most cases before the processing of large and unstructured data can start. Imputation of missing data (e.g., using the average or the value of the previous occurrence) did not lead to performance improvements of the prediction model. One reason for that could be that there is a reason why fields are empty, e.g., a certain test was not necessary or the patient was non-adherent.
The development of publicly available datasets/cohorts, such as Medical Information Mart for Intensive Care (MIMIC-III), may improve the comparability of prediction models in the future and may allow better comparisons of AUC-ROC with the literature. However, when data are strongly unbalanced and AUC-ROC is a very unreliable score, we recommend AUC-PR (precision/recall) instead.
The interdisciplinary nature of the team is a critical factor in success and is highly important to find the best solutions for all stakeholders. The team consisted of nurses, physicians in various positions, computer scientists, and medical informatics. In addition, health insurance companies and patients were involved from the beginning. An independent expert for data protection issues is expensive but a good way to find the best solution for various privacy issues.
In the future, the risk prediction model should lead to CDSS and ultimately to Automated Decision Support Systems (ADSS). However, to reach these goals, trustworthy Artificial Intelligence (AI) is necessary, for which the prediction models need to be evaluated and validated within clinical studies to fully assess their utility and benefit. Such studies should ideally be planned and executed by medical experts in close collaboration with computer scientists. Only medical doctors can assess unmet medical needs and design and perform medical trials. Medical experts should rigorously examine the usefulness and explainability of prediction models, and even inexperienced physicians need to understand the consequences and limitations of a model. Moreover, further studies are necessary to explore how such a system could be integrated into the working environment and into the existing processes to achieve the best benefit, as well as how model explanations need to be presented to further increase trust.
9.6 Discussion
We set up routine telemedicine care for adult patients after kidney and/or pancreas transplantation, including an app and telemedicine dashboard, within GDPR guidelines. The telemedicine service relies on an interoperable and standardized software architecture, which allows further modular additions in the future. The telemedicine unit aims to monitor KTRs for early detection of complications and to interact in cases of critical changes. In addition, we aim to support adherence and empower patients to improve their own healthcare [9]. We also developed risk prediction models for rejection, graft failure, and infection in adult KTRs and evaluated the models on retrospective data and within a small experimental study with medical doctors [15].
The key findings are the challenges and learnings from the implementation of a self-financed telemedicine care unit that adheres to the GDPR. At all stages of development, several new challenges had to be addressed. The main hurdles were data protection and long-term funding. Other aspects, which slowed the progress, were bureaucracy, the need for a different mindset, and the need for the same language between computer scientists and healthcare workers. However, the results of the project demonstrate the feasibility of such an interdisciplinary approach and that a human-centric approach is needed in medical informatics and its translation to clinical usefulness. For a successful telemedicine implementation, the human factor is key not only for the design of the software and dashboard but also for the user experience, the integration into the workflow, and the acceptance by the patients [6]. Finally, experienced healthcare workers have a higher impact on patient care and improve the surveillance and treatment of KTRs.
Currently, many new telemedicine approaches for medical care are under development, but with different subgroups or different endpoints, so the final proof of concept still lacks. The first question is how to determine adherence – or nonadherence – in the frame of an app-based telemedicine project. We interpret the rate of participation as adherence, thus the frequency of forwarded vital signs per week, knowing that patients, of course, measure their vital signs but forget to forward them via the app. In the second step, an algorithm was developed to calculate the actual participation and develop a tool to make the comparison easier and more reliable as a pure impression of the telemedicine team.
In the course of our work, two more projects with the aim of improving the care of KTRs in Germany should be mentioned: the project ktx360◦ (“KidneyTx360”) and a Randomized Controlled Trial (RCT) by Schmid and by Kayer et al.[18,19]. Both trials showed a positive impact of the novel treatment approaches on KTRs. ktx360◦ showed its ability to improve the adherence of patients as well as cardiovascular fitness (which was the aim of the study) [20]. In addition, hospitalization rates due to cardiovascular events were significantly lower compared to the control group, and the loss of graft function with the return to dialysis was significantly lower. Schmid and Kayer showed that telemedicine could reduce costs due to unplanned hospitalization and improve adherence within the first year after transplantation [18,19]. In addition, KTRs selected for the intervention group described a better quality of life and were more ready to return to employment. Another telemedicine project with KTRs from Australia demonstrated better adherence in the telemedicine group compared to the standard of care [21]. Here, patients were supported for 3 months and followed for 12 months. Other trials describe other telemedicine approaches [21].
During our project, it became clear that for any sustainable telemedicine solution, a clear benefit has to be demonstrated in order to convince the payer to finance such a system. Consequently, a business canvas was developed during the project, and currently we are working on a rigorous large randomized multicenter trial in Germany to demonstrate better outcomes, such as fewer hospitalizations, fewer DSA, better adherence, treatment satisfaction, and better renal function and blood pressure control.
One of the strengths of the telemedicine project is the interdisciplinary app’s design, which considers the User eXperience (UX) and the interoperable HL7 FHIR standard with an FHIR server. Another important aspect is the safe and approved digital infrastructure formally evaluated for data safety and protection. Finally, yet importantly, a contract with insurance companies allowed for continued telemedical service with a refund of expenditures (personnel, development of concepts). The weaknesses are the slow development of new features and the bugfixing process.
Over the years, the number of participants and incoming data increased; currently, three nurses and one doctor work in the telemedicine unit as opposed to one nurse and one doctor at the beginning. In addition, more tasks were implemented into the telemedicine unit, for instance, the care of KTRs with mild COVID-19 infection. However, the intense personal contacts with members of the telemedicine team aiming to support the KTRs are the driving factor behind the project. Especially since KTRs can chat via the app and can ask questions or seek help without any hurdles.
The main hurdles for the development of the prediction models were inconsistent, noisy, and imbalanced data with many missing values, which required extensive data cleansing before the model could be developed. Another problem was the lack of external, independent data sets, which limited thorough cross-validation. While better system performance might be of interest, the more interesting question to us was, how good physicians can solve the task, and is the current system already good enough to support physicians? This has been explored in the second evaluation to find out if the tool is useful and if automatic predictions are similar to those made by physicians [15].
Regarding the prediction models, motivation in the form of local and global features on why the prediction models came to a specific solution is provided along with the result of the prediction model, e.g., which laboratory values or signs and symptoms contributed to flagging a patient as critical. Before prediction models are used, their usefulness must be evaluated in the frame of studies. The first study compares the assessments of physicians and the prediction models themselves with each other. In the second step, the prediction models should be implemented into TBase, which is kept for future work. If a patient is flagged with a particular risk, the model explains why, and the telemedicine team watches the scenario develop (i.e., an observational study design). In the end, the sensitivity, specificity, and positive and negative predictive values can be assessed, making statements on the prediction models’ usefulness.
In a long-lasting process, the data were cleansed and standardized. In the primary process, where the computer scientist provides an overview of the type of data, interdisciplinary discussions were led so that the computer scientist gained a deep understanding. Transparency was ensured at all times.
As humans play an important role in evaluating the prediction models and the explainability of decisions (telemedicine team), vulnerable and marginalized groups are protected at any time.
Currently, AI prediction models make no therapeutic decisions. As explained above in detail in the first step, they must be evaluated in the frame of clinical studies before being implemented into a life system.
9.7 Conclusion
Building up a telemedicine care unit requires – independent of the subgroup – many preconditions, summarized as funding, an easy-to-modify EMR, an app, and dedicated personnel open to interaction with different disciplines. The participants need to be patient because new features need time to be released and support such digital approaches with feedback. Ultimately, however, it became obvious that the human factor is key to the successful implementation of a highly sophisticated digital platform in medicine, as this needs to be integrated into daily life for patients and into the workflow for caregivers.







