In our study of 421 acute lung injury patients, we found substantial inter-rater reliability for FCI across trained data collectors. A number of comorbidities were underdetected by the prospectively collecting data using hospital admission records versus a complete chart review. There were no significant differences in the prevalence of comorbidities detected retrospectively using discharge summaries versus complete chart review. Both the prospective and retrospective data collection methods significantly underestimated the overall FCI score compared to the gold standard method. Using multivariable regression, adjusting for age and gender, FCI scores obtained from discharge summaries explained more variance in SF-36 PFS score at 1 year than FCI score obtained from hospital admission records or complete chart review, but all three methods had similar magnitudes of association with PFS, suggesting comparative predictive value. Moreover, FCI scores explained more variance in SF-36 PFS at 1 year than the Charlson comorbidity index or the chronic health points of the APACHE II score at ICU admission.
Adjustment for patients’ baseline comorbidities is essential to understand the independent contributions of various exposures or therapies to long-term physical function and outcomes in clinical research. To the best of our knowledge, this study is the first to evaluate inter-rater reliability of FCI. Furthermore, despite both hospital admission records and discharge summaries underestimating the FCI score (as compared to the gold standard of complete chart abstraction), all three methods demonstrated similar associations with the SF-36 PFS at 1 year, suggesting comparative predictive value for this outcome. Given that the time and effort required to collect comorbidity information from a discharge summary is much more efficient than complete chart review, the use of retrospective data collection from hospital discharge summaries is a reliable and valid option for clinical research that requires comorbidity information for predicting SF-36 PFS at 1 year.
Ideally, the comorbidity information obtained from a variety of patient documentation sources should be similar, although there are likely important reasons why different data acquisition methods may vary in detection of individual comorbidities. Specifically in our study, due to the unpredictable nature of critical illness, patients admitted to ICU may have a more complete history deferred, including a review of comorbidity information (e.g., arthritis, osteoporosis, depression) unrelated to the acute illness, as the health care team initially tries to resuscitate and stabilize the patient. Thus, prospective data collection using patient records shortly after ICU admission may underdetect certain comorbidity information. Conversely, discharge summaries have the opportunity to codify the comorbidity information documented by a number of health care providers over the course of an entire hospital admission, allowing for potentially more complete or more important comorbidity to be captured.
Given the variability in the sensitivity of detecting various classes of comorbidities by data collection method, the optimal choice for data collection may vary based on the population studied, the outcome of interest, and question to be answered. This is important since prevalent comorbidities which may go undetected could result in substantial bias in the FCI score which is not accounted for
. Conversely, prospective data collection of comorbidity information may be preferred if study coordinators are already collecting other data from the medical chart at admission, and the population being studied is expected to have few pre-existing comorbid illnesses (e.g., younger patients).
The FCI is a novel and unique comorbidity scale developed with physical function as the primary outcome. In creating the FCI, the investigators hypothesized that diagnoses associated physical function would be different than those associated with mortality as used in the popular CCI
[9–13]. Thus, FCI should outperform indices designed with mortality as the outcome of interest (e.g., CCI) in predicting physical function. Our results are consistent with a previous study demonstrating that, comparing FCI to the CCI and Kaplan-Feinstein index (KFI), the FCI accounted for more variation in the SF-36 PFS, highlighting the importance of using risk models designed to predict a specific outcome.
Our study has potential limitations. While we did not explicitly evaluate the time required to collect comorbidity information by each of the three methods, it was clear that retrospective data collection using electronic discharge summaries was much less time consuming that complete chart review, and more efficient than prospective data collection from admission records, especially since it could be completed remotely using computer access to the records. Furthermore, we did not evaluate whether a hybrid method (e.g., combined prospective and retrospective data collection) would be superior to either method alone. However, the goal of our study was to demonstrate whether a more efficient method of data collection (i.e., using electronic discharge summaries as a retrospective form of data collection) would have acceptable inter-rater reliability and sufficient validity compared to the gold standard (complete chart review). Indeed, since we did collect comorbidity data both prospectively and retrospectively, the final comorbidity dataset for the ICAP parent study included comorbidities from both methods. Finally, our results were obtained from a population of ALI patients, and as such, may not be generalizable to other populations. However, the FCI has been validated in ALI patients
, and given our hypothesis that more detailed and comprehensive comorbidity information may be collected and documented in the medical record over the course of an entire hospital admission, our results are likely to be applicable to other groups of hospitalized patients.