Implementation of a machine learning application in preoperative risk assessment for hip repair surgery

Abstract

Background

This study aims to develop a machine learning-based application in a real-world medical domain to assist anesthesiologists in assessing the risk of complications in patients after a hip surgery.

Methods

Data from adult patients who underwent hip repair surgery at Chi-Mei Medical Center and its 2 branch hospitals from January 1, 2013, to March 31, 2020, were analyzed. Patients with incomplete data were excluded. A total of 22 features were included in the algorithms, including demographics, comorbidities, and major preoperative laboratory data from the database. The primary outcome was a composite of adverse events (in-hospital mortality, acute myocardial infarction, stroke, respiratory, hepatic and renal failure, and sepsis). Secondary outcomes were intensive care unit (ICU) admission and prolonged length of stay (PLOS). The data obtained were imported into 7 machine learning algorithms to predict the risk of adverse outcomes. Seventy percent of the data were randomly selected for training, leaving 30% for testing. The performances of the models were evaluated by the area under the receiver operating characteristic curve (AUROC). The optimal algorithm with the highest AUROC was used to build a web-based application, then integrated into the hospital information system (HIS) for clinical use.

Results

Data from 4,448 patients were analyzed; 102 (2.3%), 160 (3.6%), and 401 (9.0%) patients had primary composite adverse outcomes, ICU admission, and PLOS, respectively. Our optimal model had a superior performance (AUROC by DeLong test) than that of ASA-PS in predicting the primary composite outcomes (0.810 vs. 0.629, p < 0.01), ICU admission (0.835 vs. 0.692, p < 0.01), and PLOS (0.832 vs. 0.618, p < 0.01).

Conclusions

The hospital-specific machine learning model outperformed the ASA-PS in risk assessment. This web-based application gained high satisfaction from anesthesiologists after online use.

Introduction

A comprehensive preoperative evaluation can enhance the quality of patient care and is associated with a reduced mortality rate [1,2,3]. In preoperative evaluation clinics, the anesthesiologist must evaluate the patient's medical history and laboratory data, determine the patient’s physical status, and draw up a preoperative management plan in a limited time. After the initial assessment, the anesthesiologist discusses these risks with the patient and the surgical team [2]. In the case of emergency or urgent surgeries, all these tasks must be achieved with high efficiency [4].

The incidence of hip fractures is gradually increasing because the aging population is continually growing, and it is causing a heavy burden on society [5, 6]. Almost all patients with hip fractures would require surgical treatment, and anesthesia intervention is inevitable. These patients are mostly geriatric and have many comorbidities.

Several estimation tools have been used to help physicians assess operative risks [7,8,9]. These models range from simple scoring systems, such as the American Society of Anesthesiologist-Physical Status (ASA-PS) [10], to more complex calculators, such as the Surgical Risk Preoperative Assessment System (SURPAS) and the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP®) Surgical Risk Calculator [11,12,13]. Although the former is easy to use, it ignores many important parameters, such as sex, age, comorbidities, and laboratory data; while the latter includes more parameters, it requires tedious work. For instance, the ACS-NSQIP, which is an open-access web-based online tool that is currently gaining worldwide acceptance, involves manually entering 19 to 21 patient-specific variables [13,14,15]. In a busy medical ecology that requires efficiency, developing an automated preoperative evaluation system that incorporates multiple parameters is imminent.

We aimed to develop a machine learning-based application that can assist anesthesiologists in assessing specific adverse outcomes for patients required to undergo hip repair surgery. We hypothesized that a machine learning algorithm, which includes variables such as patient demographics, comorbidities, laboratory data, and anesthesiologists’ initial assessment, may have superior performance in risk assessment than the ASA-PS scoring. Through the assistance of the data-driven application, anesthesiologists will be able to effectively evaluate patients and precisely inform them of the operative risks, allowing them to have shared decision-making in a real-world medical domain.

Methods

Study design

We established a multidisciplinary team including anesthesiologists, data scientists, and information engineers for this retrospective study. Data were extracted from the Chi-Mei Medical Center’s hospital information systems (HIS) database to build the AI prediction models. We deleted the medical record number and all types of personal identification of each patient to protect their privacy.

All methods were carried out per relevant guidelines and regulations of Chi Mei Medical Center. The construction of the database was approved by the institutional review board (Serial No. 10906–008). Informed consent was waived because of the retrospective design of the study, which only involved secondary analysis of existing data and had no direct patient contact. After machine learning model training and performance testing, the optimal models were deployed into the existing HIS to assist anesthesiologists in performing preoperative risk assessment for patients with hip fractures (Fig. 1).

Fig. 1

Flow chart of study

Study setting and patient selection

Adult patients aged 20 years and above who underwent surgical treatment of a hip fracture at Chi Mei Medical Center and its 2 branch hospitals in Tainan City, Taiwan, from January 1, 2013, to March 31, 2020, were selected based on current procedural terminology (CPT) codes. The CPT codes for enrollment included hip fixation codes (27,235, 27,236, 27,244, and 27,245), hemiarthroplasty (27,125), and total hip arthroplasty (27,130) with an admission diagnosis of hip fracture (ICD-9 codes 820.x or ICD-10 codes S70-S79). Patients with incomplete perioperative data, those whose body weight was 30 kg and below, and those whose height was 100 cm and below were excluded from the study. A total of 5,301 patients were initially reviewed for this study, but only 4,448 patients were included after considering the exclusion criteria.

Feature variables

Using a similar method from previous studies [16,17,18], established clinical importance and clinical expert opinion were used to select 22 preoperative variables from the HIS dataset as inputs to the algorithm for calculating the risk of adverse events of interest. The feature variables retrieved from the HIS database include (1) patient demographics (e.g., age, sex, body mass index, and smoking status); (2) preoperative comorbidities (e.g., heart diseases such as coronary artery disease, congestive heart failure, old myocardial infarction, previous cerebral stroke, dialysis use, presence of COPD; (3) laboratory values (e.g., serum sodium, white blood cell count, hematocrit, platelet count, creatinine, blood urea nitrogen, creatinine, albumin, and prothrombin time); and (4) operative and anesthetic variables (e.g., ASA-PS status, mode of anesthesia, and the anticipated arterial line and central venous pressure monitoring). These features were integrated into the algorithm for machine learning. To develop the models, the patients were randomly split into a training cohort (70%) and a testing cohort (30%). This separation helped ensure that the test set was kept completely independent from the training set.

Study outcomes

This study’s primary outcome was a composite of postoperative adverse events, including (1) in-hospital mortality and death within 48 h after discharge (discharge death code); (2) acute stroke (ICD-9-CM codes 430 to 436 and 997.02, and ICD-10-CM codes I609, I619, I6789); (3) acute myocardial infarction (AMI) (ICD-9-CM code 410, and ICD-10-CM codes I21 or I23); (4) acute respiratory failure (ICD-9-CM codes 518.81 to 518.82, 518.84, and 518.5, and ICD-10-CM codes J96); (5) sepsis (ICD-9-CM codes 038, and ICD-10-CM codes R65); (6) acute liver failure (ICD-9-CM codes 570, and ICD-10-CM codes K7200); and (7) acute renal failure (ICD-9-CM codes 584.9 and ICD-10-CM codes 570, and ICD-10-CM codes with ICD-10-CM codes K7200); and (7) dialysis code (ICD-9-CM codes 584). Moreover, postoperative intensive care unit (ICU) admission and prolonged length of stay (PLOS) were set as secondary outcomes.

Machine learning algorithms

The models were trained with 7 machine learning algorithms consisting of (1) logistic regression, (2) random forest, (3) k nearest neighbor (KNN), (4) support vector machines (SVM), (5) light gradient boosting machine (light GBM), (6) eXtreme gradient boosting (XGBoost), and (7) deep learning of multilayer perception (MLP). To address the issue of class imbalance in the training cohort, the synthetic minority oversampling technique was utilized. Python programming language with scikit.learn machine learning library was used for model building. Grid searching with 5-fold cross-validation for hyperparameter tuning for each algorithm was performed to obtain the optimal models. The goal of the algorithms is to predict the primary and secondary outcomes.

Model performance

Each model was used to predict the test set. The specificity, sensitivity, accuracy, and area under the receiver operating characteristic curve (AUROC) were calculated and the models’ predictive performances were compared based on the AUROC value.

Implementation of web-service application to HIS

The optimal algorithm with the highest AUROC was used to build a web-based application, then integrated into the HIS for pre-anesthetic patient evaluation.

Anesthesiologist satisfaction score after AI-assisted risk assessment

After each completion of the AI-assisted risk assessment, the system automatically requests its users to grade their level of satisfaction from 0 (most dissatisfied) to 5 (highly satisfied). The first month, which was done online, was employed as the benchmark reference to compare the changes in the satisfaction of anesthesiologists during the study period from July 2020 to April 2021.

Incidence of adverse outcomes before and after web-based application deployment in HIS

We deployed the AI-assisted risk assessment application online beginning July 1, 2020. To assess whether the developed application improved the medical outcomes, we compared the incidences of primary composite adverse outcomes, ICU admission, and PLOS before (July 2019 to April 2020) and after (July 2020 to April 2021) AI-assisted risk assessment.

Statistical analysis

Descriptive statistical analysis of the data was performed using SPSS 13.0 for Windows (SPSS, Inc., Il, USA). Continuous variables were defined as the means and standard deviations or medians and ranges. Countable variables were defined with numbers and percentages. The models’ predictive performances were compared with each other and with conventional ASA-PS risk stratification based on the AUROC value using the Delong test [19]. A series of one-way analyses of variance were conducted to examine the differences in the satisfaction score among the five-month groups. After, Tukey's honestly significant difference post hoc test was performed to detect the intergroup differences. The level of significance was set at a p-value less than 0.01.

Results

Demographics

From January 1, 2013, to March 31, 2020, a total of 5,301 adult patients who had hip fractures and received hip repair surgery under GA or NA were identified. After removing excluded patients, data from 4,448 patients underwent analysis. From this, 3,114 patients (70%) were randomly allocated for training the machine learning models, and 1,334 patients (30%) were set as the validation cohort (Fig. 1).

Patient demographics and characteristics of the training and testing data sets are summarized in Table 1. The mean age of the patients was 65.3 years, and they are mostly females (57.6%). Approximately 70.8% of them were stratified as ASA-PS 3 status. The event rates were 2.3% (N = 120), 3.6% (N = 160), and 9.0% (N = 401) for composite primary adverse events, ICU admission, and PLOS, respectively. As shown in Table 1, patients with primary composite adverse events and ICU admission were older, mostly males, had anemia, longer prothrombin time, and activated partial thromboplastin time (aPTT), and had comorbidities such as chronic respiratory diseases, cancer, heart disease, dementia, or advanced chronic kidney disease (stage 4 and 5).

Table 1 Demographic data of study patients

Correlation analysis (Table 2) identified the correlation between each feature's outcome. For composite primary adverse outcomes, the most relevant features were anticipated intraoperative arterial line and central venous catheter monitoring, preoperative hemoglobin, and respiratory comorbidity; for ICU admission, the most relevant features were ASA-PS status, arterial line central venous pressure monitoring, and preoperative hemoglobin; and for PLOS, the key factors included emergency surgery, ASA-PS, central venous monitoring, ALT, eGFR, P.T., and comorbid of respiratory disease.

Table 2 The correlation coefficients between each feature and each outcome

Prediction of primary composite adverse outcomes

In the machine learning prediction of primary composite adverse outcomes such as in-hospital mortality, mortality within 48 h after discharge, and major organ injury, the sensitivity by logistic regression, SVM, and lightGBM all reached 0.710; the SVM had the highest specificity (0.716) followed by KNN (0.711) (Table 3). All models, except KNN, achieved high AUROCs which were between 0.734 (XGBoost, 95% C.I.: 0.636 ~ 0.831) and 0.794 (Logistic regression, 95% C.I.: 0.718 ~ 0.869) (Appendix 1).

Table 3 Predictive performance of machine learning algorithms on primary composite adverse outcomes^*

Prediction of postoperative ICU admission

As shown in Table 4, in the prediction of ICU admission, MLP (0.812) and logistic regression (0.792) had the highest sensitivity. Further, logistic regression (0.791), lightGBM (0.769), and the random forest (0.760) had the highest specificity. Moreover, logistic regression, lightGBM, and the random forest had the highest accuracy (between 0.760 to 0.791). Except for KNN and SVM, all models had AUROCs between 0.825 (XGBoost, 95% C.I.: 0.772 ~ 0.878) and 0.856 (Logistic regression, 95% C.I.: 0.804 ~ 0.908) (Appendix 2).

Table 4 Predictive performance of machine learning algorithms on ICU admission^*

Prediction of prolonged length-of-stay (PLOS)

The results of model prediction on PLOS are shown in Table 5. The random forest and lightGBM had higher sensitivity (0.783 and 0.767, respectively) and specificity (0.783 and 0.774, respectively) than other algorithms. Moreover, the random forest had the best performance with the highest AUROCs (0.854; 95% C.: 0.818 ~ 0.890) (Table 5 and Appendix 3).

Table 5 Predictive performance of machine learning algorithms on prolonged length-of-stay*

The performance of machine learning models and ASA-PS

Based on the results, an AI web-based application was constructed using logistic regression for adverse outcomes and ICU admission, and the random forest for PLOS. Figure 2 shows a snapshot of the AI web service application for predicting adverse outcomes in the pre-anesthetic visit clinic.

Fig. 2

A Snapshot of the web-based application in the hospital information system

The results in Table 6 indicate that the machine learning AI web-based application had superior AUROC scores (Delong test, P < 0.001) than ASA-PS stratification in terms of primary composite adverse outcomes (0.776 vs. 0.629), ICU admission (0.844 vs. 0.629), and PLOS (0.854 vs. 0.618).

Table 6 Comparison of AI models with ASA-PS for primary composite adverse outcomes, ICU admission and prolonged length of hospital stay

The incidences of adverse outcomes before and after AI-assisted application deployment

Table 7 demonstrates the demographics and incidences of adverse outcomes in 545 and 500 patients before and after implementing the online web-based application. There was no statistically significant decrease in the incidence of primary composite adverse events (3.3 vs. 1.6%, p = 0.117) or ICU admission (4.4 vs. 2.4%, p = 0.109) after the web-based application was initially employed for clinical use.

Table 7 The incidences of major adverse outcomes before and after AI web-based application online use

The satisfaction score web-based application from anesthesiologists

The AI Assist Application was launched on July 1, 2020, and by April 30, 2021, a total of 500 patients were evaluated under the assistance of AI. Figure 3 illustrates that the satisfaction score rose from 3.21 ± 0.51 (1^st month online) to 4.70 ± 0.56 (10^th month). The score was significantly higher starting in the 4^th month after the application was launched (P < 0.01).

Fig. 3

Anesthesiologists’ satisfaction ratings of the web-based application since its implementation

Discussion

In this retrospective study using the HIS database, machine learning methods were applied in our hospital-specific real-world medical domain to assist anesthesiologists in their preoperative risk assessment for patients required to undergo hip fracture repair surgery in terms of primary composite adverse outcomes (mortality and major organ injuries), the need for ICU admission and PLOS. The newly developed AI assist application was found to have significantly higher sensitivity, specificity, accuracy, and performance (AUROC) than that of the ASA-PS, the traditional and most widely used risk stratification method. The major strength of this study is its successful integration of the AI-assisted app into the hospital’s HIS system. The novel contribution of this study is that the machine learning algorithm empowered the ASA-PS scoring, allowing more specific prognostic assessments for patients undergoing hip surgery. Moreover, this online app is user-friendly and received high satisfaction scores from anesthesiologists who used it.

Machine learning can simultaneously deal with numerous variables by building statistical models based on outliers and nonlinear interactions among variable [20, 21]. To use our application, the anesthesiologist, after evaluating the patient, first inputs the ASA-PS followed by the mode of anesthesia and whether an arterial line or a ventral venous catheter is anticipated. Then, the AI assist application automatically captures 22 features, which are important independent risk factors [22, 23], from the HIS. All the anesthesiologist has to do is click the calculation button. The application will start to run and calculate the risk scores for in-hospital mortality, ICU admission, and PLOS. The results of the calculation are then displayed on the computer screen.

The results of this study demonstrated that modern AI computer systems can not only collect and display data but can also play an active role in assisting physicians with their risk assessments, allowing them to make a shared decision with the patient or their family members.

Previous machine learning applications in hip fracture have demonstrated high potential for automated detection of hip fractures on radiographs and hip fracture risk prediction [24,25,26]. Further, recent studies have tried to build a precise model for mortality of hip fracture surgery [27]. This current research not only developed an AI-based risk predictive model that performed better than ASA-PS in terms of risk assessment for hip fracture surgery but also successfully incorporated the application into the hospital’s existing HIS, allowing it to be used in daily practice. It is important for hospitals to establish a more reliable and available model of risk assessment for patients who need to undergo hip fracture surgery. A precise model could help improve physicians’ shared decision-making with their patients and assist in evaluating the need for critical care monitoring after surgery. Machine learning techniques can integrate a large amount of data already captured in the HIS, which offers a prediction model with better predictive performance and facilitates automation.

This research does not suggest the discontinuation of the ASA-PS system nor does it refute the need for human intelligence; instead, it aims to add a machine-learning algorithm to facilitate efficiency in preoperative risk assessment. Anesthesiologists need to judge the patient’s ASA physical status and decide whether an arterial line or CVC is anticipated perioperatively. The AI assist application takes the above information, in conjunction with patient data captured from the HIS, to calculate the risk of adverse events.

Machine learning algorithms have been proven to more accurately assess the risks associated with anesthesia and surgery [28, 29]. A study published by Ehlers et al. [28] used the insurance claims database and calculated the Naïve Bayes algorithm to predict the risk of postoperative complications and showed superiority to Charlson's comorbidity index. In the present study, more variables were adapted, and the data were used for the training of 7 algorithms. A recent study by Li et al. [30] reported AI prediction using the random forest algorithm to predict 1-year mortality after hip repair surgery. They collected data from 1,330 and 744 patients to train and validate the AI algorithm, respectively. In the present study, data from 3,114 patients for training and 1,334 for validation were included. Because the sample size is greater, the statistical power is therefore stronger. Moreover, other than mortality, the risk of ICU admission and PLOS were also estimated.

Aside from the widely accepted ASA-PS system, there are also some pre-anesthetic risk stratification tools being used in hospitals, such as the ACS-NSQUIP, an open-access online tool based on the logistic model [31]. Our AI application shares some similarities with ACS-NSQIP. Both tools have some variables in common, such as age, sex, ASA, emergency surgery, and CPT procedure code. The ACS-NSQIP is calculated based on logistic regression, while our AI application used seven algorithms, including logistic regression, for machine learning. After comparing the AUROC of these algorithms, the best algorithm was selected to build the AI prediction tool. Moreover, although there is now an open-excess ACS-NSQIP, it has not been built into the Chi Mei Medical Center HIS system yet. Therefore, the present study did not include this as a reference comparison. Further comparative studies are very worth conducting in the future.

Although anesthesiologists have affirmed the online application of this study's app, we still have not observed a significant effect on reducing the incidence of adverse outcomes, ICU admission, and PLOS. It may require a longer observation time and a larger population of patients to justify the efficacy of this web-based application.

Limitations

Some limitations are inherent in retrospective machine learning projects using hospital-specific databases. First, the accuracy of prediction algorithms at specific hospitals may be limited by hospital-specific factors. However, the methodology could theoretically be generalized to similar hospitals with similar patient races or under similar health insurance systems. Second, this study was dependent on the correctness of the ICD-9- or ICD-10-CM coding while identifying study cases, comorbidities, and complications. These codes were reviewed and validated by auditors of medical records for the insurance system to ensure the accuracy of the claims; however, there is still the possibility of miscoding and misclassifying some diseases and conditions. Third, the study’s data were extracted from a single medical institution and its 2 branch hospitals; thus, an underlying referral bias might have existed. Therefore, to obtain a more generalizable result, external validation using patient cohorts from other institutions is required. Fourth, since this is a retrospective study, further validation in a prospective manner to demonstrate predictive capability is needed. Fifth, postoperative mortality in the study’s model was limitedly captured from in-hospital complications and in-hospital death or death within 48 h after discharge. Our models demonstrated relatively short-term mortality or adverse events because some patients, especially those who had complications or were dissatisfied with the surgical service, might have transferred to other hospitals without a referral. Therefore, the study endpoints were limited to the in-hospital period. Sixth, non-geriatric hip fracture may have different pathophysiologic mechanisms from geriatric hip fractures and may require different assessment tool. In our preliminary analysis, we subgrouped study patients into those above 50 (n = 3551) and below 50 years (n = 897), however, this number of non-geriatric patients was insufficient to support machine learning. Therefore, the current research cannot meet this need.

Conclusions

The AI assist application developed using a machine learning algorithm was found to be helpful for anesthesiologists in evaluating the risks associated with hip surgery more efficiently and accurately than the traditional ASA-PS stratification method. Although this study may be limited by hospital-specific factors, it could still be generalized to hospitals with similar patient races and comparable health insurance systems. Moreover, this web-based application gained a high satisfaction score from anesthesiologists, which implies an urgent need for automated artificial intelligence assistance in preoperative risk assessment.