Skip to main content

Problems of subgroup analysis in randomized controlled trial

Abstract

Multiple subgroup analyses of the same data increase the risk of generating false positive findings. All outcomes and planned subgroup analyses should thus be prespecified and described in the original trial registry. When outcome changes during an ongoing trial seem justifiable, publications must disclose and explain such changes.

Peer Review reports

To the Editor:

The publication by Licker et al. [1] in BMC Anesthesiology raises several concerns. Foremost, all of the reported patients were included in a previous publication by the same group of authors [2]. The principal findings of both publications and of another very recent publication by the same group of authors [3] are basically identical, that is, glucose-insulin-potassium (GIK) infusion before start of cardiopulmonary bypass (CPB) improves LV function after open heart surgery. This creates the impression that three independent investigations [1,2,3] have documented a benefit of GIK-infusion during on-pump cardiac surgery. However, the publication by Licker et al. [1] reports findings in a post hoc defined subgroup of patients already included and analyzed in the first publication assessing the identical outcome, although the title states that this is a secondary analysis of the previously reported randomized controlled trial [2].

The authors claim that analysis of this subset of patients was pre-planned. This is an incorrect statement because it fundamentally contradicts the trial registration which reads: “We will investigate the impact of short term GIK on the extent of myocardial injuries as well as on the left-ventricular systolic and diastolic function in 2 high-risk groups of cardiac surgical patients: Patients with cardiac dysfunction undergoing aortocoronary bypass surgery and patients with severe aortic stenosis.” [4]. Furthermore, the authors define secondary study endpoints as” other TEE parameters as well as hemodynamic parameters”. However, the trial registration [4] lists as secondary outcome measures intraoperative systolic and diastolic cardiac function using transesophageal echocardiography (TEE), intraoperative hypo- and hyperglycemia and hypo- and hyperkalemia, 48 h postoperative serious cardiovascular adverse events (myocardial infarction, cardiac arrhythmia, low cardiac output, stroke), and intraoperative and 48 h postoperative serum troponin and creatine kinase concentrations. As a general principle, whenever changes from the original trial registration occur, they must be fully explained and reported to allow appropriate interpretation of results [5].

The authors neither mention nor critically discuss the considerable implications of a discrepancy between pre-specified and reported primary outcome measures. Accurate reporting of a pre-specified primary outcome in a subsequent publication is a critical component of clinical research, and incorrect reporting constitutes a major scientific flaw [6, 7]. Outcome misreporting increases the likelihood that reported differences are chance findings or are exaggerated [8]. Therefore, all outcomes and planned subgroup analyses should uniformly be prespecified and described in the original trial registry, and confidence intervals be provided for all outcomes to indicate the precision (uncertainty) of the estimate [9, 10]. The CONSORT (Consolidated Standards of Reporting Trials) statement even discourages, in general, multiple subgroup analyses of the same data because of the increased risk of generating false positive findings [7]. Unfortunately, reported primary outcomes in published randomized controlled trials frequently differ from those specified in the respective clinical trial registry [11, 12]. This includes publications in journals publicly endorsing the CONSORT guidelines on best practice in trial reporting which emphasize the importance of reporting all pre-specified outcomes [7, 12].

In the original publication [2], the a priori power calculation based on the primary outcome postcardiotomy ventricular dysfunction (PCVD) required 88 patients per group for adequate statistical power. Of those 88 patients, the present publication [1] reports selective findings in just 54 and 38 patients, respectively. In the absence of an a priori power calculation, it cannot be ruled out that the study is statistically underpowered. A statistical significance of interaction test was listed as one of 10 critical criteria for assessment of the credibility of a subgroup effect [9].

The CONSORT flow diagram was designed for the reporting of randomized trials [7]. Inclusion of such flow diagram in the publication [1] makes the reader believe that isolated coronary artery bypass graft (CABG) surgery, the combination of CABG surgery and aortic valve replacement (AVR), and poor quality or no TEE were exclusion criteria as per protocol. As this was not the case, presentation of a CONSORT flow diagram is inappropriate in this context. It is equally inappropriate and misleading to start the Discussion section with the words, “In this randomized controlled trial ( …)” when this publication does definitely not report findings of a randomized controlled trial.

Several of the statistically significant differences between groups in the absolute values of TEE variables reported in the publication [1] lie within the accuracy and precision of the applied methodology and are thus of questionable clinical relevance. Furthermore, presenting the same data as percent changes is unwarranted duplication of identical data and exaggerates the observed small changes in absolute values. Interpretation is exclusively based on p-values, a practice that is increasingly being discouraged [5, 6, 9]. Confidence intervals are not provided.

In conclusion, the present publication [1] is based on a previous study by the same authors [2]. It substantially overlaps in study hypothesis, study population, methods, data and conclusions, does not adhere to the pre-specified outcomes as defined in the trial registry, and does not introduce novel scientific aspects. Editors and reviewers can limit such shortcomings by, (a) ensuring that all results of an investigation are reported in a single comprehensive publication, (b) routinely consulting the trial registry when assessing a manuscript, and (c) demanding that authors report and analyze pre-specified outcomes [9, 10]. When outcome changes during an ongoing trial seem justifiable, publications must disclose and explain such changes [11].

Availability of data and materials

Not applicable.

Abbreviations

LV:

left ventricular

GIK:

glucose-insulin-potassium

CPB:

cardiopulmonary bypass

TEE:

transesophageal echocardiography

CONSORT:

Consolidated Standards of Reporting Trials

PCVD:

postcardiotomy ventricular dysfunction

CABG:

coronary artery bypass graft; aortic valve replacement

References

  1. 1.

    Licker M, Diaper J, Sologashvili T, Ellenberger C. Glucose-insulin-potassium improves left ventricular performances after aortic valve replacement: a secondary analysis of a randomized controlled trial. BMC Anesthesiol. 2019;19(1):175.

    Article  CAS  Google Scholar 

  2. 2.

    Ellenberger C, Sologashvili T, Kreienbuhl L, Cikirikcioglu M, Diaper J, Licker M. Myocardial protection by glucose-insulin-potassium in moderate- to high-risk patients undergoing elective on-pump cardiac surgery: a randomized controlled trial. Anesth Analg. 2018;126(4):1133–41.

    Article  CAS  Google Scholar 

  3. 3.

    Licker M, Reynaud T, Garofano N, Sologashvili T, Diaper J, Ellenberger C. Pretreatment with glucose-insulin-potassium improves ventricular performances after coronary artery bypass surgery: a randomized controlled trial. J Clin Monit Comput. 2020;34(1):29–40.

    Article  Google Scholar 

  4. 4.

    https://clinicaltrials.gov/ct2/show/NCT00788242. Accessed 21 Jan 2020.

  5. 5.

    http://www.consort-statement.org/checklists/view/32%2D%2Dconsort-2010/73-changes-to-trial-design. Accessed 21 Jan 2020.

  6. 6.

    Priebe HJ. Pre-specified outcomes must be followed. J Clin Monit Comput. 2020;34(1):187–8.

    Article  Google Scholar 

  7. 7.

    Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. BMJ. 2010;340:c332.

    Article  Google Scholar 

  8. 8.

    Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, Williamson PR. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365.

    Article  Google Scholar 

  9. 9.

    Sun X, Briel M, Buss JW, You JJ, Akl EA, Mejza F, Bala MM, et al. Credibility of claims of subgroup effect in randomized controlled trials: systematic review. Br Med J. 2012;344:e1553.

    Article  Google Scholar 

  10. 10.

    Sun X, Ioannidis JP, Agoritsas T, et al. How to use a subgroup analysis: users' guide to the medical literature. JAMA. 2014;311:405.

    Article  CAS  Google Scholar 

  11. 11.

    Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med. 2015;13:282.

    Article  Google Scholar 

  12. 12.

    Goldacre B, Drysdale H, Dale A, Milosevic I, Slade E, Hartley P, Marston C, Powell-Smith A, Heneghan C, Mahtani KR. COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials. 2019;20:118.

    Article  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

The author declares that he has not received any institutional/hospital/departmental funds or funds from any other funding agency/source. Open access funding provided by Projekt DEAL.

Author information

Affiliations

Authors

Contributions

HJP is the only author. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Hans-Joachim Priebe.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The author declares that he has no competing interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Priebe, H. Problems of subgroup analysis in randomized controlled trial. BMC Anesthesiol 20, 186 (2020). https://doi.org/10.1186/s12871-020-01105-8

Download citation

Keywords

  • Trial registration
  • Outcome reporting
  • CONSORT guideline