- Research article
- Open Access
- Open Peer Review
Perioperative blood transfusions and survival in patients with non-small cell lung cancer: a retrospective study
© Cata et al.; licensee BioMed Central Ltd. 2013
- Received: 18 April 2013
- Accepted: 13 November 2013
- Published: 15 November 2013
Perioperative blood transfusions have been associated with poor clinical outcomes in the context of oncological surgery. Current literature is inconclusive whether blood transfusions are linked to shorter recurrence free and overall survival after lung cancer surgery. We hypothesize that blood transfusions in patients undergoing surgery for non-small cell lung cancer are associated with poor oncological survival.
After IRB approval, perioperative data from 636 patients who underwent lung cancer surgery was collected. Patients were evaluated for time to tumor recurrence and overall survival.
60 patients were transfused and 576 subjects were not. Patients who received transfusion were more likely to have more advanced disease (p = 0.018), and preoperative low hemoglobin concentrations (p < 0.0001) compared to non-transfused patients. In the multivariable Cox regression analysis, blood transfusion was associated with a significant reduction in recurrence free survival (p = 0.025), HR: 1.55 (95% CI: 1.06-2.27) and overall survival (p = 0.0002) HR: 2.04 (95% CI: 1.41-2.97). However, analysis after propensity score matching between the two groups revealed that the effect of blood transfusion was significant for reduction in overall survival (p = 0.0356), HR: 1.838 (95% CI: 1.04-3.22) but not for recurrence free survival (p = 0.1460), HR: 1.493 (95% CI: 0.87-2.56).
Perioperative administration of red blood cells appears be associated with a decreased overall survival but not recurrence free survival after lung cancer surgery. Our study has the limitations of a retrospective review. Hence, our results should be confirmed by a prospective randomized control trial.
- Overall Survival
- Blood Transfusion
- Propensity Score
- Recurrence Free Survival
- Preoperative Anemia
Surgery remains the most effective treatment for stage I to IIIa non-small cell lung cancer (NSCLC), which is the second most common cancer in the United States . Although the rate of perioperative blood transfusions in patients with NSCLC is relatively low, the number of patients who will receive packed red blood cells (pRBCs) across the world during surgery for NSCLC is not insignificant and hence their exposition to adverse effects of blood transfusions [2, 3]. Unfortunately, administration of pRBCs is associated with a distinct pathology of immunosuppression, known as transfusion related immune suppression (TRIM). TRIM has been linked to a number of complications including effects on cancer recurrence . TRIM is considered to contribute to progression of minimal residual disease (MRD) to clinical metastasis particularly in patients who do not receive postoperative chemo-radiation as adjuvant therapy after surgery [5–9]. Several mechanisms have been described to be responsible of TRIM. The accumulation of tumor growth factors, lysophophatidylcholines and pro-tumoral cytokines in the units of pRBCs along with a predominant immunosuppressive response of the host in reaction to microparticles and fragmented RBCs facilitate the growth of cancer cells . Moreover, patients undergoing lung cancer surgery have significant impairment in their ability to attack cancer cells as the result of cancer itself, surgery, anesthetics and analgesics given intra- and postoperatively . Thus, TRIM may further impair the innate immunity of these patients after surgery.
The results of a Cochrane review indicate that blood transfusions are associated with an increased risk of cancer recurrence in patients with colorectal cancer . Several retrospective studies in patients with NSCLC demonstrated that perioperative blood transfusions were associated with poorer disease-free and overall survival [2, 13–15]. However, other studies have concluded to the contrary [16, 17]. The published retrospective studies had several limitations including small sample size, differences in the surgical procedures performed, differences in the type of blood products administered, differences in the follow-up duration and the presence of confounding tumor and treatment related factors [14, 15, 17]. Thus, we decided to investigate whether the administration of pRBCs to patients with stage 1–3 NSCLC who underwent tumor resection at MD Anderson Cancer Center had an effect on their recurrence-free survival and overall survival.
After approval from MD Anderson Cancer Center institutional review board, we recorded information from 636 patients with stage 1-3a NSCLC cancer who underwent lobectomies, sublobar resection, sleeve lobectomy, or pneumonectomy from 2004 through 2006. Patients who received allogenic pRBCs within 30 days before surgery and during their postoperative hospitalization were compared with those who received no transfusion. We collected demographic (age, gender, body mass index and ASA physical status), co-morbidities (smoking, alcohol consumption, diabetes mellitus, coronary artery disease, creatinine higher than 2 mg/dL, stroke, and preopeative anemia defined as hemoglobin lower than 12 g/dL), tumor-related variables (tumor size and stage) and treatment-related variables (preoperative and postoperative chemotherapy and radiation). We also recorded the number of pRBCs units transfused.
The primary endpoint of interest was recurrence free survival (RFS), defined as the time in months from surgery to recurrence or death, whichever event occurred first. Overall survival (OS) was defined as the time in months from surgery to death from any cause. The exposure variable of interest was blood transfusion status (yes or no) for surgery. Other prognostic variables of interest included age at surgery, gender, BMI, ASA, cancer stage, tumor size, smoking status, alcohol consumption, patient comorbidities, preoperative hemoglobin level, preoperative and postoperative chemoradiation.
Propensity score analysis was undertaken in an attempt to adjust for potential bias associated with factors related to the decision to undergo blood transfusion. This statistical methodology has often been used in observational studies to control for nonrandom treatment assignment of patients by adjusting for differences in covariates between the treatment groups. To control for factors that may confound the relationship between blood transfusion and RFS or OS, we determined the propensity score to receive blood transfusion for each patient, using multivariable logistic regression. The covariates included in the analysis were: age at surgery, gender (F or M), BMI, ASA (2, 3, or 4), stage (1, 2, or 3), and preoperative hemoglobin level. Given the propensity scores for all patients, we identified sets of patients, one transfused patient randomly matched with two who did not undergo blood transfusion, using a 5-to-1 digit greedy match algorithm. The differences in propensity scores in each set were no more than 0.06. We used absolute standardized differences to assess balance in the baseline variables between patients who underwent blood transfusion and those who did not undergo blood transfusion in the matched cohort. The absolute standardized differences for all baseline covariates were< 9% in the matched cohort.
For the prematching cohort, differences in demographic, clinical, and tumor characteristics between groups of patients who underwent blood transfusion and those who did not undergo blood transfusion were compared by using Fisher’s exact test or Chi-square test for categorical variables and Wilcoxon rank sum test for continuous variables. Kaplan-Meier method was used to evaluate the effect of patient characteristics on RFS and OS. Multivariable Cox proportional hazards models were used to determine the effect of blood transfusion on RFS and OS after adjusting for the prognostic variables.
Patient and tumor characteristics by blood transfusion status before and after matching
Hb g/dL, (SD)
We used a total of 636 patients in our analysis, 60 in the blood transfusion group, and 576 in the no-tranfusion group. The median follow-up time for the entire study cohort was 5.3 years for the censored observations. The median follow-up time was 5.3 years among patients who received blood transfusions and 5.3 years among patients who did not undergo have blood transfusions. Table 1 lists the characteristics of patients in the blood transfusion and no-transfusion groups before and after propensity score matching.
Prior to matching, patients who received blood transfusions differed from those who did not have transfusion in terms of important prognostic factors. Patients who received transfusion were more likely to have stage II to III disease (p = 0.018) compared to patients who did not undergo transfusion. The transfused patients were also more likely to have low hemoglobin levels (p < 0.0001). Matching using the propensity score substantially reduced the imbalance. None of the variables was significantly different between the transfused patients and non-transfused patients in the matched cohort.
Recurrence free survival
Recurrence free survival (RFS) by patient characteristics
Median RFS time in months (95% CI)
RFS rate at 3 years (95% CI)
RFS rate at 5 years (95% CI)
> = 66
Hb < 12 g/dL
Number of units
0.67 (0.63 0.71)
Association between blood transfusion (Y vs. N) and recurrence free survival
Unadjusted (univariate analysis), before matching (E/N = 277/599)
Adjusted for covariates (multivariate analysis)* (E/N = 277/595)
Propensity-matched, stratifying on the matched pair (E/N = 78/135)
Overall survival (OS) by patient characteristics
Median OS time in months (95% CI)
OS rate at 3 years (95% CI)
OS rate at 5 years (95% CI)
> = 66
Hb < 12 g/dL
Number of units
Association between blood transfusion (Y vs. N) and overall survival
Unadjusted (univariate analysis), before matching (E/N = 243/636)
Adjusted for covariates (multivariate analysis)* (E/N = 243/632)
Propensity-matched, stratifying on the matched pair (E/N = 68/135)
In colorectal cancer patients, a metaanalysis by Amato and Pescatori demonstrated that blood transfusions are an independent factor for cancer recurrence . In our work, we found that perioperative blood transfusions in the context of NSCLC surgery are associated with shorter OS; however, this association was not observed for RFS. Our findings are not fully in line with some of the results published by other authors [2, 14, 15, 18–20].
Pena et al. demonstrated that perioperative blood transfusions did not increase the risk of cancer recurrence or worsen overall survival after lung cancer resection . Similarly, Berardi et al., and Panagopoulos et al. reported that perioperative blood transfusions were not an independent risk factor for poor oncological outcomes [22, 23]. Finally, a retrospective study that included 493 patients also found no association between blood transfusions and overall survival after NSCLC cancer surgery . Contrary to those studies and similar to our findings in terms of OS, 5 retrospectives analysis have shown that blood transfusion are associated with shorter OS [2, 13–15, 25]. Importantly, a recent manuscript published by Churchhouse et al. summarizes the current evidence on this topic and highlights the controversy between blood transfusion and risk of cancer recurrence after lung cancer surgery .
There are several reasons that might explain these controversial findings. First, the presence of unknown confounding factors. Second, the indications for blood transfusions and the type of lung resection; in particular the latter (degree and complexity of surgical procedure) has been shown to have a significant impact on recurrence-free survival . Third, it has been suggested that the use of blood transfusions as a surrogate for correction of low hemoglobin values (anemia) may have contributed to the differences amongst the published studies on this topic. It may very well be that preoperative anemia and the myelo-recovery from the disease process and therapeutic interventions may in fact be the more important and determining prognostic factor . And fourth, the statistical methodology used to analysis the data. In this regards, our study is particularly different to the rest of the published studies because we performed a propensity score matching analysis and we consider this a strength of the present work. Propensity matching score analysis has become the suggested method to analyze observational data, in particular in the context of perioperative medicine, because the technique allows estimating the treatment effect controlling for covariates such as comorbidities. In our study, by matching on the propensity score, we eliminated those transfused patients with no comparable non-transfused subjects, hence eliminating potential bias in factors contributing to the decision for perioperative transfusion . In our study, the standardized differences for all covariates were lower than 15% in the post-matching cohort, suggesting substantial reduction of bias between the two groups.
Our univariate analysis also demonstrated that preoperative Hb was associated with poor RFS and OS. It has been suggested that the degree of anemia is a marker of myelosuppression in patients receiving chemoradiation, and in fact, the severity of anemia may be correlated with tumor response and hence an important predictor of survival . Paradoxically, low hemoglobin concentrations may cause tumoral hypoxia which can trigger adaptive mechanisms that may alter the phenotype of the cancer cells turning them more aggressive . Several studies, even those that did not find an association between blood transfusion and NSCLC recurrence, have observed that preoperative anemia is a risk factor for recurrence and it has been hypothesized that blood transfusions may only represent an intervention to correct low hemoglobin concentrations [22, 23]. In fact, patients whose hemoglobin levels remained lower than 12 g/dl despite transfusion had worse prognosis, thus suggesting that the degree of anemia before transfusions, may be a predictor of survival . This concept is supported by a study suggesting that after adjusting for preoperative anemia, blood transfusion is an independent factor for recurrence and overall survival in patients with NSCLC stage 1 cancer . In our study, transfused patients had significantly lower preoperative Hb concentration; hence we decided to include this variable in the matching in order to avoid significant confounding.
Another interesting finding of our study is that the number of blood transfusions received appeared to be associated with poor outcomes. There are several possible reasons that might explain our findings. First, a “dose-dependent” effect; although a our findings are not in line with those reported by Keller et al., it is possible to speculate that the magnitude of immunosuppression associated with blood transfusions is greater as the number of units transfused increases; hence, the likelihood to recur after surgery . Second, patients with larger tumors and complex procedures who are subject to a higher chances of incomplete tumor resections and postoperative complications may be also exposed to a higher number of units transfused, hence, our findings only a reflection of the magnitude of bulk tumoral disease .
Our study has several limitations. First, this is a retrospective study, hence there are unknown factors that may affect the studied outcomes and were not captured in our data collection (i.e. tumor markers, age of the transfused units, postoperative complications and cause of death). Postoperative complications after major thoracic surgery are known to negatively impact mortality, hence, it is possible to speculate that in our database those patients transfused might have suffered from more transfusion-related or unrelated postoperative complications which would have significantly affected their survival . We were not able to capture the cause of death of all our patients, hence our results might have been heavily confounded by the presence of comorbidities known to have significant impact on OS. In fact, our univariate and multivariate analysis indicated a potential association between ASA physical status and OS; hence, since co-morbidities are part of the ASA score, we included it in the matching analysis.
Second, we included a heterogeneous group of patients in regards to the stage of the disease. Four other studies that analyzed recurrence and mortality in an uniform group of patients (stage 1) have shown that blood transfusions are in fact an independent factor for poor outcomes [2, 18, 23, 25]. Interestingly, Ng et al. suggested that the impact of blood transfusion is less evident in patients with advanced stages of NSCLC because the poor baseline prognosis of this patients . Third, a major limitation of the present work is that we included patients who had received both leucoreduced and non-leucoreduced pRBCs during the perioperative period. The impact of leukoreduced transfusions on NSCLC recurrence and patient survival is controversial. Ng et al. demonstrated that the administration of leukodepleted pRBCs was associated with poorer outcomes than non-transfused patients . Similar results were shown by Panagopoulos et al. suggesting that at least for NSCLC transfusion of pRBCs itself rather than the leukoreduced status of the unit has a more significant impact on outcomes . Blood units given intraoperatively in our institution are typically non-leucoreduced, while the units dispensed to the patient floors are non-leukoreduced. Thus a given patient with multiple perioperative blood transfused may have received leucoreduced units pre- or postoperatively and non-leucoreduced blood intraoperatively. We have therefore not analyzed patients according to the leucoreduction status of the blood units received, as it would not have reflected the current transfusion practice at our institution. Finally, our conclusion of no significant effect of pRBC transfusions on RFS after propensity matching should be studied further in a larger sample size since the Cox regression analysis showed an association between blood transfusion and increased cancer recurrence. It is possible that the small sample size in the matching cohort group may have contributed to our results.
The perioperative administration of pRBCs appears not to be associated with shorter disease-free survival but with worse overall survival after NSCLC surgery. A randomized controlled trial is needed to support our findings and to elucidate the real impact of perioperative blood transfusion on cancer recurrence and survival after lung cancer surgery.
- Siegel R, Naishadham D, Jemal A: Cancer statistics, 2012. CA: Cancer J Clin. 2012, 62 (1): 10-29. 10.3322/caac.20138.Google Scholar
- Nosotti M, Rebulla P, Riccardi D, Baisi A, Bellaviti N, Rosso L, Santambrogio L: Correlation between perioperative blood transfusion and prognosis of patients subjected to surgery for stage I lung cancer. Chest. 2003, 124 (1): 102-107. 10.1378/chest.124.1.102.View ArticlePubMedGoogle Scholar
- Gwin JL, Keller SM: Blood transfusion practices after resection of intrathoracic neoplasms. J Surg Oncol. 1993, 54 (1): 34-37. 10.1002/jso.2930540110.View ArticlePubMedGoogle Scholar
- Churchhouse AM, Mathews TJ, McBride OM, Dunning J: Does blood transfusion increase the chance of recurrence in patients undergoing surgery for lung cancer?. Interac Cardiovasc Thorac Surg. 2012, 14 (1): 85-90. 10.1093/icvts/ivr025.View ArticleGoogle Scholar
- Funaki S, Sawabata N, Nakagiri T, Shintani Y, Inoue M, Kadota Y, Minami M, Okumura M: Novel approach for detection of isolated tumor cells in pulmonary vein using negative selection method: morphological classification and clinical implications. Eur J Cardiothorac Surg. 2011, 40 (2): 322-327.PubMedGoogle Scholar
- Sawabata N, Okumura M, Utsumi T, Inoue M, Shiono H, Minami M, Nishida T, Sawa Y: Circulating tumor cells in peripheral blood caused by surgical manipulation of non-small-cell lung cancer: pilot study using an immunocytology method. Gen Thorac Cardiovasc Surg. 2007, 55 (5): 189-192. 10.1007/s11748-007-0101-2.View ArticlePubMedGoogle Scholar
- Brunsvig PF, Flatmark K, Aamdal S, Hoifodt H, Le H, Jakobsen E, Sandstad B, Fodstad O: Bone marrow micrometastases in advanced stage non-small cell lung carcinoma patients. Lung Cancer. 2008, 61 (2): 170-176. 10.1016/j.lungcan.2007.12.018.View ArticlePubMedGoogle Scholar
- Sawabata N, Ohta M, Matsumura A, Nakagawa K, Hirano H, Maeda H, Matsuda H: Optimal distance of malignant negative margin in excision of nonsmall cell lung cancer: a multicenter prospective study. Ann Thorac Surg. 2004, 77 (2): 415-420. 10.1016/S0003-4975(03)01511-X.View ArticlePubMedGoogle Scholar
- Felip E, Rosell R, Maestre JA, Rodriguez-Paniagua JM, Moran T, Astudillo J, Alonso G, Borro JM, Gonzalez-Larriba JL, Torres A, et al: Preoperative chemotherapy plus surgery versus surgery plus adjuvant chemotherapy versus surgery alone in early-stage non-small-cell lung cancer. J Clin Oncol. 2010, 28 (19): 3138-3145. 10.1200/JCO.2009.27.6204.View ArticlePubMedGoogle Scholar
- Cata JP, Wang H, Gottumukkala V, Reuben J, Sessler DI: Inflammatory response, immunosuppression, and cancer recurrence after perioperative blood transfusions. British J Anaesth. 2013, 110 (5): 690-701. 10.1093/bja/aet068.View ArticleGoogle Scholar
- Cata JP, Bauer M, Sokari T, Ramirez MF, Mason D, Plautz G, Kurz A: Effects of surgery, general anesthesia, and perioperative epidural analgesia on the immune function of patients with non-small cell lung cancer. J Clinical Anesth. 2013, 25 (4): 255-262. 10.1016/j.jclinane.2012.12.007.View ArticleGoogle Scholar
- Amato A, Pescatori M: Perioperative blood transfusions for the recurrence of colorectal cancer. Cochrane Database Syst Rev. 2006, 1: CD005033Google Scholar
- Ng T, Ryder BA, Chern H, Sellke FW, Machan JT, Harrington DT, Cioffi WG: Leukocyte-depleted blood transfusion is associated with decreased survival in resected early-stage lung cancer. J Thorac Cardiovasc Surg. 2012, 143 (4): 815-819. 10.1016/j.jtcvs.2011.12.031.View ArticlePubMedGoogle Scholar
- Moores DW, Piantadosi S, McKneally MF: Effect of perioperative blood transfusion on outcome in patients with surgically resected lung cancer. Annals Thorac Surg. 1989, 47 (3): 346-351. 10.1016/0003-4975(89)90371-8.View ArticleGoogle Scholar
- Piantadosi S, Moores DW, McKneally MF: The adverse effect of perioperative blood transfusion in lung cancer. Chest. 1994, 106 (6 Suppl): 382S-384S.View ArticlePubMedGoogle Scholar
- Keller SM, Groshen S, Kaiser LR: Blood transfusion and lung cancer recurrence. Ann Thorac Surg. 1989, 48 (5): 746-747.View ArticlePubMedGoogle Scholar
- Thomas P, Michelet P, Barlesi F, Thirion X, Doddoli C, Giudicelli R, Fuentes P: Impact of blood transfusions on outcome after pneumonectomy for thoracic malignancies. European Respir J: Off J European Soc Clinical Respir Physiol. 2007, 29 (3): 565-570.View ArticleGoogle Scholar
- Tartter PI, Burrows L, Kirschner P: Perioperative blood transfusion adversely affects prognosis after resection of stage I (subset N0) non-oat cell lung cancer. J Thorac Cardiovasc Surg. 1984, 88 (5 Pt 1): 659-662.PubMedGoogle Scholar
- Little A, Wu H, Ferguson M, Ho C, Bowers V, Segalin A, Staszek V: Perioperative blood transfusion adversely affects prognosis of patients with stage I non-small-cell lung cancer. Am J Surg. 1990, 160 (6): 630-10.1016/S0002-9610(05)80762-7.View ArticlePubMedGoogle Scholar
- Hyman N, Foster R, DeMeules J, Costanza M: Blood transfusions and survival after lung cancer resection. Am J Surg. 1985, 149 (4): 502-507. 10.1016/S0002-9610(85)80047-7.View ArticlePubMedGoogle Scholar
- Pena CM, Rice TW, Ahmad M, Medendorp SV: Significance of perioperative blood transfusions in patients undergoing resection of stage I and II non-small-cell lung cancers. Chest. 1992, 102 (1): 84-88. 10.1378/chest.102.1.84.View ArticlePubMedGoogle Scholar
- Berardi R, Brunelli A, Tamburrano T, Verdecchia L, Onofri A, Zuccatosta L, Gasparini S, Santinelli A, Scartozzi M, Valeri G, et al: Perioperative anemia and blood transfusions as prognostic factors in patients undergoing resection for non-small cell lung cancers. Lung Cancer. 2005, 49 (3): 371-376. 10.1016/j.lungcan.2005.04.011.View ArticlePubMedGoogle Scholar
- Panagopoulos ND, Karakantza M, Koletsis E, Apostolakis E, Sakellaropoulos GC, Filos KS, Eleni T, Dougenis D: Influence of blood transfusions and preoperative anemia on long-term survival in patients operated for non-small cell lung cancer. Lung Cancer. 2008, 62 (2): 273-280. 10.1016/j.lungcan.2008.02.025.View ArticlePubMedGoogle Scholar
- Rzyman W, Dziadziuszko R, Skokowski J, Wilimski R, Raiter A, Szymanowska A, Jassem J: The influence of blood transfusion on survival in operated non-small cell lung cancer patients. J Thorac Cardiovasc Surg. 2003, 126 (3): 755-760. 10.1016/S0022-5223(03)00217-4.View ArticlePubMedGoogle Scholar
- Little AG, Wu HS, Ferguson MK, Ho CH, Bowers VD, Segalin A, Staszek VM: Perioperative blood transfusion adversely affects prognosis of patients with stage I non-small-cell lung cancer. Am J Surg. 1990, 160 (6): 630-632. 10.1016/S0002-9610(05)80762-7. discussion 633View ArticlePubMedGoogle Scholar
- Ginsberg RJ, Rubinstein LV: Randomized trial of lobectomy versus limited resection for T1 N0 non-small cell lung cancer. Lung cancer study group. Ann Thorac Surg. 1995, 60 (3): 615-622. 10.1016/0003-4975(95)00537-U. discussion 622–613View ArticlePubMedGoogle Scholar
- Zanutto EL: A comparison of propensity score and linear regression analysis of complex survery data. J Data Sci. 2006, 4: 67-91.Google Scholar
- Kishida Y, Hirose T, Shirai T, Sugiyama T, Kusumoto S, Yamaoka T, Okuda K, Adachi M, Nakamura A: Myelosuppression induced by concurrent chemoradiotherapy as a prognostic factor for patients with locally advanced non-small cell lung cancer. Oncol Lett. 2011, 2 (5): 949-955.PubMedPubMed CentralGoogle Scholar
- Semenza GL: HIF-1 and tumor progression: pathophysiology and therapeutics. Trends Mol Med. 2002, 8 (4 Suppl): S62-S67.View ArticlePubMedGoogle Scholar
- Keller SM, Groshen S, Martini N, Kaiser LR: Blood transfusion and lung cancer recurrence. Cancer. 1988, 62 (3): 606-610. 10.1002/1097-0142(19880801)62:3<606::AID-CNCR2820620327>3.0.CO;2-Q.View ArticlePubMedGoogle Scholar
- Tabutin M, Couraud S, Guibert B, Mulsant P, Souquet PJ, Tronc F: Completion pneumonectomy in patients with cancer: postoperative survival and mortality factors. J Thorac Oncol: Off Publ Int Assoc Study Lung Cancer. 2012, 7 (10): 1556-1562.View ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2253/13/42/prepub
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.