The Scottish intensive care society audit group (SICSAG) database
SICSAG has maintained a national database of patients admitted to adult, general critical care units in Scotland since 1995. Currently, all adult, general and specialist intensive care and combined intensive care/high dependency units (critical care units) in Scotland participate voluntarily in the audit. Data are collected prospectively using a dedicated software system. Annual data extracts are pooled centrally onto servers at the Information Services Division and validation queries relating to discharges, outcomes, ages and missing treatment information are then issued and fed back to individual units for checking by local and regional audit coordinators.
This study was approved by the Privacy Advisory Committee, NHS National Services Scotland (application number 53/10).
Inclusion and exclusion criteria
Data were extracted from the SICSAG database for all admissions to all 24 adult, general critical care units in Scotland between 1 January 2007 and 31 December 2009. During the study period, specialist cardiothoracic critical care units were not participating in the national audit; admissions to one specialist neurocritical care unit were not included in the data extract. The following admissions were excluded from the analysis: admissions flagged in the database as ‘Exclude from severity of illness scoring’; readmissions of the same patient within the same acute hospital stay; admissions missing the outcome of acute hospital mortality; admissions missing age, location prior to admission or primary reason for admission to the critical care unit; and admissions for whom the primary reason for admission was unable to be mapped onto the ICNARC Coding Method (see below).
The ICNARC model
The ICNARC model was developed and validated using data from the ICNARC Case Mix Programme [7, 8]. Risk predictions are calculated for each admission based on the following predictors:
age in years at admission to the critical care unit;
location prior to admission to the critical care unit and urgency of surgery;
cardiopulmonary resuscitation within 24 hours prior to admission to the critical care unit;
ICNARC Physiology Score – an integer score between 0 and 100 based on derangement in 12 physiological parameters during the first 24 hours following admission to the critical care unit;
primary reason for admission to the critical care unit; and
interactions between the ICNARC Physiology Score and primary reason for admission.
The ICNARC model is regularly recalibrated to Case Mix Programme data to ensure accurate, contemporaneous comparative audit for the Case Mix Programme. The most appropriate recalibration was selected based on the time period of data included in the analysis – this was a recalibration undertaken in 2009 using Case Mix Programme data from 194,892 admissions to 187 critical care units between 1 January 2006 and 31 December 2008.
In order to apply the ICNARC model to data from the SICSAG database, certain assumptions and recoding were required, detailed below. After applying this recoding, the predicted risk of acute hospital mortality from the ICNARC model was calculated for each admission using standard algorithms developed for the Case Mix Programme.
Location prior to admission
In the ICNARC model, for admissions to the critical care unit from an imaging department and those from the recovery area (not for postoperative use but when used as a temporary critical care area), the previous location is used to assign a weight. For admissions collected to Version 0 of the SICSAG dataset (phased out from June 2008 to May 2009), only a single location immediately prior to the critical care unit was recorded and therefore the weightings for location prior to admission for these admissions was assigned based on the most common previous location in both SICSAG Version 203 data (introduced from June 2008) and Case Mix Programme data. Admissions from an imaging department were assumed to have previously been in an emergency department and admissions from the recovery area were assumed to have previously been on a general ward.
Systolic blood pressure
In the ICNARC Physiology Score, weighting of the systolic blood pressure (SBP) is based on the lowest value during the first 24 hours following admission to the critical care unit. For SICSAG data (all Versions), only the highest SBP with paired diastolic blood pressure (DBP) and the lowest DBP with paired SBP were recorded. The lowest SBP was therefore imputed using a regression model fitted to 574,864 admissions to 181 critical care units in the Case Mix Programme between 1995 and 2008 with all these parameters recorded. The resulting equation was:
In the ICNARC Physiology Score, weighting of arterial pH is based on the lowest pH during the first 24 hours following admission to the critical care unit. For SICSAG data (all Versions), only the pH from the arterial blood gas with the lowest partial pressure of oxygen (PaO2) was recorded. The lowest pH was therefore imputed using a regression model fitted to 1,011,217 admissions to 224 critical care units in the Case Mix Programme between 1995 and 2013 with both pH measurements recorded. The resulting equation was:
In the ICNARC Physiology score, weighting of neurological status is based on either the lowest Glasgow Coma Score during the first 24 hours following admission to the critical care unit (for admissions not sedated during that entire period) or a separate weighting for patients that were sedated or paralysed and sedated during the first 24 hours. For admissions collected to Version 203 of the SICSAG dataset (introduced from June 2008), sedation was not recorded. Admissions were therefore assumed to be sedated if they had no lowest Glasgow Coma Score recorded during the first 24 hours following admission to the critical care unit (this was true for 99% of such admissions in SICSAG Version 0 data).
Primary reason for admission
In the ICNARC model, weighting of the primary reason for admission to the critical care unit is based on weightings for conditions/body systems from the ICNARC Coding Method . The ICNARC Coding Method is a five-tier, hierarchical system for coding reasons for admission to critical care that contains 795 individual conditions within a hierarchy of type (surgical or non-surgical), body system, anatomical site, pathological or physiological process and individual condition. Coding to the system tier is sufficient to be able to assign a weight for the ICNARC model, although all admissions in the Case Mix Programme are coded to at least the site tier. For all SICSAG data, the primary reason for admission to the critical care unit was collected using Scottish Intensive Care Society (SICS) diagnostic coding. These diagnoses were mapped to appropriate codes within the ICNARC Coding Method by a consultant intensivist with extensive experience of coding data for the Case Mix Programme. Of the 423 SICS diagnoses in use, 295 (70%) were mapped to a specific condition in the ICNARC Coding Method, 44 (10%) were mapped to the process tier of the hierarchy, 37 (9%) to the site tier, 28 (7%) to the system tier, and 19 (4%) were unable to be mapped (see Additional file 1).
The APACHE II model
The Acute Physiology And Chronic Health Evaluation (APACHE) II model was selected as a comparator for this study as it was the model in use in Scotland at that time. The SICSAG database does not include all the requisite fields to enable a head-to-head comparison against other, more recent, risk prediction models. The APACHE II model was originally developed using data from 19 critical care units in 13 US hospitals , and has subsequently been validated and recalibrated using UK data [6, 11]. Risk predictions are calculated for each admission based on the following predictors:
the APACHE II Score – an integer score between 0 and 71 comprising an Acute Physiology Score (0–60 points) based on derangement in 12 physiological parameters during the first 24 hours following admission to the critical care unit, age points (0–6) for age categories of ≤44, 45–54, 55–64, 65–74 or ≥75 years, and chronic health points (0–5) for very severe conditions in the past medical history;
admission to the critical care unit following emergency surgery; and
diagnostic categories based on the primary reason for admission to the critical care unit.
Values of predicted acute hospital mortality were supplied by the Information Services Division, calculated from the original published coefficients  using the standard algorithms applied for routine reporting of the SICSAG audit results at that time.
The ICNARC model was validated using measures of calibration, discrimination and overall fit, as described below. The validation was conducted in the full three-year SICSAG database extract and for each year separately.
Discrimination was assessed by the c index , which is equivalent to the area under the receiver operating characteristic (ROC) curve . Calibration was assessed graphically and tested using the Hosmer-Lemeshow test for perfect calibration in ten equal sized groups by predicted probability of survival . As the Hosmer-Lemeshow test does not provide a measure of the magnitude of miscalibration and is very sensitive to sample size , calibration was also assessed using Cox’s calibration regression, which assesses the degree of linear miscalibration by fitting a logistic regression of observed survival on the predicted log odds of survival from the risk model . Accuracy was assessed by Brier’s score (the mean squared error between outcome and prediction)  and Shapiro’s R (the geometric mean of the probability assigned to the event that occurred) , and the associated approximate R-squared statistics (termed the ‘sum-of-squares’ R-squared and the ‘entropy-based’ R-squared, respectively), which are obtained by scaling each measure relative to the value achieved from a null model .
The performance of the ICNARC model was compared with that of the APACHE II model. The difference in c index between the two models was assessed using the method of DeLong et al. . Confidence intervals for observed acute hospital mortality were calculated using the method of Wilson .
All statistical analyses were performed using Stata/SE Version 13.0 (StataCorp LP, College Station, Texas, USA).