Short- and long-term mortality prediction in critically ill subjects: a cohort study

Background. There are several scoring systems used for in-hospital mortality prediction in critical illness. Their application in a local scenario requires validation to ensure appropriate diagnostic accuracy. Also, their use in assessing post-discharge mortality in the ICU survivors has not been extensively studied. Aim. To evaluate the ability of APACHE II, III and SAPS II to predict in-hospital and post-discharge mortality in adult ICU patients. Material and methods. APACHE II, APACHE III and SAPS II, with corresponding predicted mortality ratios, were calculated for 303 consecutive patients admitted to the 10-bed ICU in 2016. Long-term mortality was calculated based on information taken from PESEL database. Results. Median APACHE II, APACHE III and SAPS II scores were 19, 67 and 44 points, with corresponding in-hospital mortality ratios of 28.1, 18.5 and 34.8%. Observed in-hospital mortality was 35.6%. 12-month post-discharge mortality reached 17.4%. All systems predicted in-hospital mortality (p<0.05): APACHE II (AUC=0.783; 95%CI 0.732-0.828), APACHE III (AUC=0.793; 95%CI 0.743-0.838) and SAPS II (AUC=0.792; 95%CI 0.742-0.836), as well as mortality after ICU discharge (p<0.05): APACHE II (AUC=0.712; 95%CI 0.643-0.775), APACHE III (AUC=0.721; 95%CI 0.653-0.783) and SAPS II (AUC=0.695; 95%CI 0.625-0.759), with no statistically significant difference between them (p>0.05). Conclusions. Although the predictive values were the highest for APACHE III and SAPS II, no differences were noticed between the scores. In case of post-discharge mortality, diagnostic accuracy was much lower. Further studies are needed to create scores estimating the long-term prognosis of subjects successfully discharged from the ICU.


Background
The main goal of the admission to the intensive care unit (ICU) is to reduce morbidity-related complications and therefore, to prevent mortality due to possibly reversible severe deterioration in clinical condition of the patient. Several simple, acknowledged tools are commonly used for outcome prediction in critical illness. Those scorings are based on the worst data obtained within the first 24 hours post-admission and they are not recalculated during the stay. The higher scores are reached, the higher in-hospital mortality risk is. First models estimating the risk of in-hospital death were developed over 30 years ago. The first two are the Acute Physiology and Chronic Health Evaluation (APACHE) system (1981) (1) and the Simplified Acute Physiology Score (SAPS) (1988) (2). Since then, many attempts were made to improve their diagnostic accuracy, and subsequent versions, including APACHE II, III, IV and SAPS II and III (3)(4)(5)(6)(7) were developed.
Although APACHE system was created basing on data from the U.S. hospitals only, whereas SAPS relied on data from Europe and North America, the first tool is more commonly used in clinical research. Although all up-to-date versions of both scoring systems were verified in terms of their diagnostic accuracy, older scoring models (i.e. APACHE II, SAPS II) remain 'gold standards' in prognostication among severely ill patients in individual ICUs worldwide. Both APACHE and SAPS performance in predicting in-hospital mortality has already been verified in patients with various diagnoses (8)(9)(10)(11)(12)(13)(14)(15)(16). Their use in assessing post-discharge mortality in the ICU survivors has not been studied so often (17)(18)(19)(20)(21).
One ought to remember that usability of those systems in the management of individual patients remains limited. They were primarily developed for outcome prediction and therefore they can be used only to compare the ICU performance and in quality improvement initiatives. Their application in a local ICU setting requires validation to ensure appropriate diagnostic accuracy.
We therefore sought to evaluate the ability of three scoring systems, i.e. APACHE II and III, and SAPS II to predict in-hospital and post-discharge mortality in adult patients at the tertiary ICU.

Material And Methods
An observational prospective study was performed at the 10-bed mixed university ICU in Poland. The study covered 303 consecutive patients admitted between 01.01.2016 and 31.12.2016. Readmissions (n = 7) were excluded from analysis. Data including demographics and comorbidities were recorded from medical records. Clinical and laboratory data were recorded on admission. Physiological data was recorded in 1-hour periods. Data was always collected and verified independently by two researchers. Appropriate scorings of APACHE II, APACHE III and SAPS II, and their corresponding predicted mortality ratios were calculated. Additionally, for each patient the clinical background of 4 admission (i.e. surgical/ medial; in-hospital/ out-of-hospital) and the outcome were assessed.
In-hospital mortality was defined as a death occurring during index hospitalization, regardless of the time of hospital stay. Post-discharge mortality was verified based on information acquired from PESEL Their distribution was verified with Shapiro-Wilk test. Chi-square or Fisher's exact test were applied for qualitative variables. All tests were two-tailed. Receiver operating characteristic (ROC) curves were drawn and the areas under the ROC curves (AUROC) were calculated to assess predictive value of the scoring systems on mortality. A 'p' value < 0.05 was considered statistically significant.
The median scorings of APACHE II, APACHE III and SAPS II were: 19, 67 and 44 points, respectively ( Fig. 1). The corresponding predicted in-hospital mortality ratios were: 28.1%, 18.5% and 34.8%, respectively, with observed-to-predicted mortality rates of 1.27, 1.92 and 1.02, respectively. Table II presents mortality indices with regard to the source of admission. In-hospital mortality was statistically significantly higher for medical than surgical patients. It was also higher in patients admitted from another hospital or from the emergency room (i.e. out-of-hospital admissions) 5 compared to subjects transferred within our hospital from another wards (i.e. in-hospital admissions).
Similar findings were observed in relation to post-discharge mortality.
All three investigated systems predicted in-hospital mortality with good diagnostic accuracy (AUC of 0.8) (Fig. 2). Although the AUC was the highest for APACHE III, no statistically significant differences were noticed between the scores. Sub-analyses revealed that APACHE III score was the most powerful in predicting early mortality among surgical subjects (AUC = 0.845; 95%CI 0.780-0.897; p < 0.001), whereas SAPS II had the highest accuracy for outcome prediction among patients admitted from the out-of-hospital settings (AUC = 0.738; 95%CI 0.570-0.867; p = 0.008) ( Table III).
All three investigated tools predicted 12-month post-discharge mortality in the studied population in a statistically significant way, however their diagnostic accuracy was much lower (AUC of ~ 0.7) (Fig. 3).
APACHE III was also the most powerful in predicting post-discharge mortality among surgical patients (AUC = 0.726; 95%CI 0.639 to 0.801; p < 0.0001) but APACHE II score was the most accurate in medical patients (AUC = 0.688; 95%CI 0.531 to 0.819; p < 0.001). The scoring systems failed to predict post-discharge mortality in patients after abdominal surgeries and in those admitted from the out-of-hospital settings in a statistically significant way (Table III).

Discussion
This single-center study aimed to investigate the ability of the three commonly used scoring systems to predict mortality in critically ill patients. We revealed that although all of them were of comparable accuracy in predicting in-hospital mortality, either APACHE III or SAPS II should be recommended as the first-choice tool. The usefulness of the systems in 12-month outcome prediction in the ICU survivors proved to be limited.
We found that in-hospital ICU mortality rate was 35.6%. which was relatively high compared to international data, but lower than the value observed in the Silesia region (43.7%) (22). Higher mortality in Polish ICUs compared to other European countries (23) was under debate in recent years and is rather due to differences in patient populations. indications for ICU admission, availability of ICU beds and organization of end-of-life care in Poland. This is also due to the skeptical attitude of some practitioners regarding guidelines on futile therapy (24,25) and official ICU admission criteria (26). Patients admitted to Polish ICUs are more often at higher risk of death compared with other countries, but ICU mortality observed in the Silesian Registry of Intensive Care Units was lower than that predicted by the APACHE II scoring system (27).
In our study, APACHE II, APACHE III and SAPS II scores and the predicted ICU mortality were as follows: 19 points (i.e. mortality rate of 28.1%). 67 (mortality rate of 18.5%) and 44 points (mortality rate of 34.8%). respectively. In case of all three studied scales expected mortality was lower than observed.
The cause of this phenomenon appears to be complex, it may result from substantial differences between the patient's population in our unit (mixed admissions. including post-operative cases in the first priority) and target populations these prognostic models were developed for. Medical patients were confirmed to have higher mortality than surgical patients, which is in line with previous research on this issue (28).
The reliability of the data collected is important because poor source data quality, the number and type of missing physiological variables can influence the mortality predictions. In the original APACHE II study variables were missing in 13% of the cases (29). In our data series 14% of variables were missing in total in all three studied scoring systems that should be taken into account in data interpretation. The process of data collection is burdened with high risk of bias. In case of APACHE II score it was observed that the main causes of data errors are inconsistent choice between highest and lowest values and problems with GCS score determination in sedated patients (29). We used the pre-sedation GCS in sedated patients if available, data was always verified by two members of the study team independently.
Two main objective criteria are used for prognostic scales performance evaluation: calibration and discrimination. Calibration refers to how closely the estimated probabilities of mortality correlate with the observed mortality and is of great importance for clinical trials or comparison of care between ICUs. Discrimination refers to the ability of a prognostic score to classify patients as survivors or nonsurvivors and is measured by ROC curves (i.e. AUC). In our study all three investigated systems predicted in-hospital mortality with good diagnostic accuracy (AUC of ~ 0.8), with no statistically significant differences between them. Our observations remain consistent with the previous studies proving high accuracy of the scoring systems (28,(30)(31)(32). The most powerful tool was APACHE III (AUC = 0.793) together with SAPS II (AUC = 0.792). SAPS II seems to perform better in our clinical setting as its observed-to-predicted mortality rate was 1.  (32). Differences in the performance of scoring systems might result from variation in case mix. standards. structure and organization of medical care. lifestyles and genetic differences between populations (7). Therefore, despite numerous studies performed so far on this subject, there is still a need to validate these prognostic models using data of independent samples from different ICUs in different countries or even regions.
In the present study APACHE III score was the most powerful in predicting early mortality among surgical ICU patients. Surgical patients have better survival prognosis than medical ICU patients (6,30). The explanation of this fact is quite simple: in these patients the reason for ICU admission is mostly their unstable condition resulting from the performed long-lasting extensive surgical procedure, and not as much from their poor general condition prior the surgery or their comorbidities.
All three investigated systems predicted 12-month post-discharge mortality in a statistically significant way, however their diagnostic accuracy was much lower (AUC of ~ 0.7). In a study by such an application further studies should be conducted to create scores estimating the long-term prediction. In this context one ought to bear in mind that proper screening and accurate identification of patients who will stay at risk after their successful discharge from the ICU might be of great importance to avoid ICU readmissions, further deterioration of quality of life and higher postdischarge mortality.
The present study has some limitations. As a single-center study, there may be bias with regard to heterogeneous population and relatively small sample size. The final results in the scoring systems may be affected by the confounding effect of the data selection process and Glasgow Coma Scale results calculation. The follow up period in our study was limited to 12 months after the date of ICU admission. Finally, we did not include SOFA score into our analysis. However, that scoring system was primarily created for prognostication among septic patients so seems less universal in the mixed ICU setting than APACHE or SAPS (35).

Conclusions
Although all systems have good diagnostic accuracy in predicting in-hospital mortality in the ICU patients, either APACHE III or SAPS II should be recommended as the most powerful tools. Their usefulness to predict the outcome after 12 months post-ICU discharge is limited, so further studies are needed to build mathematical equations estimating the long-term prognosis of subjects 13 successfully discharged from the ICU.

Ethics approval and consent to participate
Patient confidentiality was ensured as the dataset was fully anonymized. Under Section 21 and 22 of the Act of 5 December 1996 on the Medical Profession (Poland), due to the non-interventional design of the study, no approval of the Ethics Committee was required.

Availability of data and materials
The datasets used and analysed during the current study are available from the corresponding author on reasonable request  In-hospital mortality prediction by APACHE II and III. and SAPS III scores 20 Figure 3 Post-discharge mortality prediction by APACHE II and III. and SAPS III scores