The quality of reporting of RCTs used within a postoperative pain management meta-analysis, using the CONSORT statement
© Borg Debono et al.; licensee BioMed Central Ltd. 2012
Received: 30 January 2012
Accepted: 4 July 2012
Published: 4 July 2012
Randomized controlled trials (RCTs) are routinely used in systematic reviews and meta-analyses that help inform healthcare and policy decision making. The proper reporting of RCTs is important because it acts as a proxy for health care providers and researchers to appraise the quality of the methodology, conduct and analysis of an RCT. The aims of this study are to analyse the overall quality of reporting in 23 RCTs that were used in a meta-analysis by assessing 3 key methodological items, and to determine factors associated with high quality of reporting. It is hypothesized that studies with larger sample sizes, that have funding reported, that are published in journals with a higher impact factor and that are in journals that have adopted or endorsed the CONSORT statement will be associated with better overall quality of reporting and reporting of key methodological items.
We systematically reviewed RCTs used within an anesthesiology related post-operative pain management meta-analysis. We included all of the 23 RCTs used, all of which were parallel design that addressed the use of femoral nerve block in improving outcomes after total knee arthroplasty. Data abstraction was done independently by two reviewers. The two main outcomes were: 1) 15 point overall quality of reporting score (OQRS) based on the Consolidated Standards for Reporting Trials (CONSORT) and 2) 3 point key methodological item score (KMIS) based on allocation concealment, blinding and intention-to-treat analysis.
Twenty-three RCTs were included. The median OQRS was 9.0 (Interquartile Range = 3). A multivariable regression analysis did not show any significant association between OQRS or KMIS and our four predictor variables hypothesized to improve reporting. The direction and magnitude of our results when compared to similar studies suggest that the sample size and impact factor are associated with improved key methodological item reporting.
The quality of reporting of RCTs used within an anesthesia related meta-analysis is poor to moderate. The information gained from this study should be used by journals to register the urgency for RCTs to be clear and transparent in reporting to help make literature accessible and comparable.
Transparent and well designed, conducted and reported randomised controlled trials (RCTs) are the gold standard in evaluating healthcare interventions . The quality of reporting of RCTs in publications is important because it acts as a proxy for health care providers and researchers to appraise the quality of the methodology, conduct and analysis expected of an RCT [2–6]. The evaluation of the quality of reporting can be compromised when RCTs lack methodological rigour, thus generating biased results possibly resulting in exaggeration or shrinking of treatment effects [1, 7–14]. Poor conduct in the quality of reporting of RCTs may allow the introduction of treatments that are less effective than the studies themselves report . Hence, it is important to know the quality of RCTs used in meta-analyses in order to identify the reliability of the evidence that may inform treatment and care in medicine, including treatment in the realm of anesthesiology and pain management [4, 7, 12, 15, 16].
Readers and reviewers of published RCTs need complete, clearly written, and transparent information on a study’s methodology and findings in order to assess its quality . Attempts in assessing the quality of reporting recurrently fail because authors of RCTs neglect to provide comprehensive and clear descriptions of critical methodological information [1, 17–19]. In response to inadequate reporting the Consolidated Standards of Reporting Trials (CONSORT) group created the CONSORT statement and published its latest revision of the statement in 2010 [1, 6]. The CONSORT 2010 Statement includes a 25 item checklist and a flow diagram. These are recommended items which should be incorporated into an RCT [1, 6]. The statement provides guidance for reporting all RCTs with a focus on individually randomised, two groups, parallel trials .
Following the CONSORT initial statement publication in 1996, there has been improvement in the quality of reporting of some RCTs [1, 10, 20–22]. However, despite improvements noted, there continues to be observational studies reporting the inadequate quality of reporting of many RCTs [1, 17], including RCTs in specialized journals [1, 21, 23–25]. There are a few other studies that have investigated the quality of reporting of RCTs specifically in anesthesia specialized journals. Many of them have evaluated the quality of reporting using items similar to items on the CONSORT 2010 checklist [7, 26–28]. Some of the specialty journals in anesthesiology have also adopted or supported the CONSORT statement, yet the quality of reporting of parallel-group RCTs still remains poor . Improvements made in the quality of reporting in the last decade may indicate a demand on the “quality, consistency and transparency”  of RCT reporting as well as endorsement of the CONSORT statement by medical journals which publish clinical trials [1, 26]. The quality of reporting of RCTs may be improved in specialty journals pertaining to anesthesiology by clearly illustrating methodology. Areas in methodology requiring attention are allocation concealment (randomization blinding), blinding of data collectors and “discussion of Type II error in negative trials” .
We have two primary aims for this study. The first aim of this study is to analyse the overall quality of reporting and reporting of 3 key methodological items of 23 RCTs used in a meta-analysis that investigated analgesia outcomes in RCTs comparing femoral nerve block (FNB) with epidural and patient-controlled analgesia (PCA) after total knee arthroplasty . Secondly, we aim to determine factors associated with higher quality of reporting. We hypothesized that studies with larger sample sizes, that have funding reported, that are published in journals with a higher impact factor and that are in journals that have adopted, endorsed or supported the CONSORT statement will be associated with better overall quality of reporting and reporting of the 3 key methodological items. Our study is unique in that we evaluated the quality of reporting of RCTs used within a meta-analysis using 15 items plus 3 methodologically related items, all of which were directly from the CONSORT 2010 checklist. It has been suggested that the quality of studies included in meta-analyses should be regularly evaluated as studies show that inadequate quality of trials may distort the results of systematic reviews and meta-analyses . We consider it important to do an update on the status of the quality of reporting of RCTs published within anesthesia specialized journals as this will contribute more evidence around the quality of reporting of RCTs used. Much of our study’s method was modelled upon a previous study assessing the quality of reporting of RCTs in general endocrinology literature .
The 23 RCTs included in this study for quality of reporting and key methodological item evaluation came from a meta-analysis that compared FNB, with and without sciatic nerve block, with epidural and PCA after total knee arthroplasty . This meta-analysis included 23 RCTs for its own analysis. It was of interest to this study to know the reporting quality and reporting of key methodology items included in this meta-analysis. All 23 RCTs assessed were parallel-design and addressed the issue of comparative treatment effectiveness with respect to morphine consumption, incidence of side effects and pain experiences, which was scored on a pain scale both at rest and with activity .
Data abstraction process
An electronic Excel standardized data collection form was used to extract data from each of the 23 RCTs. One pair of trained reviewers (VBD and SZ), both of whom were blinded to each other’s ratings, abstracted data independently. This form was pilot tested and subsequently modified for final revision. Any disagreement between the reviewers was rectified by consensus and chance-adjusted inter-rater agreements were calculated.
The rating of overall reporting quality
Description of overall quality of reporting items
1. Title or Abstract
Identification as a randomized trial in the title and a structured summary of the trial design, methods, results, and conclusions.
Scientific background and explanation of rationale.
Specific objectives or hypotheses.
Eligibility criteria for participants. Settings and locations where the data were collected
The interventions for each group with sufficient details to allow replication, including how and when they were actually administered.
Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed.
7. Sample Size
How sample size was determined. When applicable, explanation of any interim analyses and stopping guidelines.
8. Randomization: Sequence Generation
Method used to generate the random allocation sequence Type of randomisation; details of any restriction (such as blocking and block size).
9. Randomization: Implementation
There is mention of: Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions?
10. Statistical Methods
Statistical methods used to compare groups for primary and secondary outcomes. Methods for additional analyses, such as subgroup analyses and adjusted analyses.
11. Participants Flow
For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome. For each group, losses and exclusions after randomisation, together with reasons.
Dates defining the periods of recruitment and follow-up. Why the trial ended or was stopped.
13. Baseline Data
A table showing baseline demographic and clinical characteristics for each group.
14. Outcomes and Estimates
For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval). For binary outcomes, presentation of both absolute and relative effect sizes is recommended.
All important harms or unintended effects in each group
16. Ethical Issues
The approval of an ethics committee and obtaining of informed consent from participants are stated
A flowchart of participants in each stage of the RCT (randomization, allocation, follow-up, and analysis for primary outcome) is provided
Rating of key methodology items
Appropriate concealment of allocation mechanisms, blinding, and numbers analysed (intention-to-treat principle) were assessed separately because they are highly important in avoiding bias and distortions of the effect estimates [4, 9, 26, 31–35]. The methodological items are also underreported in studies with a high overall reporting quality score hence emphasizing the need to assess them separately [34, 35].
Appropriate concealment of allocation is a process that assigns patients in an RCT into alternative treatment groups in a way that will prevent the people who manage the allocation of patients as well as the patients themselves from foreknowledge of the allocation mechanism. This is done in order to avoid selection bias . Obvious and easily manipulated variables such as medical record numbers or birthdays are not adequate for use in allocation concealment mechanisms. In order to avoid selection bias, allocation concealment mechanisms considered appropriate by the Cochrane Handbook are centralized randomization schemes, numbered, coded vehicles and sequentially numbered, sealed and opaque envelopes . Hence, we considered allocation concealment to be appropriate in this study if centralized randomization, numbered, coded vehicles, or opaque, sealed, and sequentially numbered envelopes were reported .
Blinding refers to the process by which study participants, health care providers, outcome assessors, data assessors and manuscript writers are kept unaware of the intervention to which patients are randomized. Blinding helps to reduce the risk of knowing which intervention was received and this is critical, as knowledge of the assigned intervention can affect outcomes and assessments of outcomes [8, 36]. Physicians and textbooks vary in their understanding and definitions of single, double, and triple blinding thus making the sole use of these terms in a study ambiguous in describing the blinding method. Despite this, single, double and triple blinding are terms commonly and singularly used to report blinding status without the accompaniment of descriptors indicating who was indeed blinded [36–38]. A meta-analysis indicated that double blinding is associated with potential bias reduction; double blinded studies reported a 15% lower treatment effect than non-blinded studies . Sometimes there are cases where the blinding of patients and health care providers is not feasible. This is especially true in instances where there is a comparison of the effectiveness of a minimally invasive procedure with a procedure that is more invasive, as with surgery. In this case, patients or investigators cannot easily be blinded . In these instances, where blinding is not feasible, we considered blinding to have been done appropriately if at least one specific group was unambiguously reported as blinded . In RCTs where blinding did not have any barriers, at least two groups had to be unambiguously reported as blinded to fulfill this methodological item criteria [3, 8, 38].
We defined the numbers analyzed based on the intention-to-treat (ITT) principle. In this study we define ITT analysis to include all participants randomized into a trial and its analysis regardless if treatment or the intervention was administered, if patients satisfied the entry criteria, if patients withdrew from the study, or if there were deviations from the protocol [8, 39–41]. Though the meaning of ITT has been interpreted differently by numerous RCTs, the Cochrane Handbook has compiled evidence indicating that the definition we have used prevents biased treatment effects and is a pragmatic way to understand the effects of a clinical intervention [41, 42].
Each of the three above mentioned methodological items was scored 1 point if the method was appropriately reported and performed and 0 points if it was inappropriate or ambiguous. A combined methodological score of these three items was calculated for each trial by adding the scores of each item with a possible score ranging from 0 to 3 inclusively. This score was termed Key Methodological Item Score (KMIS). An inter-rater agreement was calculated for each of the three included methodological items as done in a previous study .
Definition of predictor variables
Sample size was defined as the number of patients randomized in each trial. The journal impact factor refers to an index number assigned and calculated by Thomson Reuters to a particular journal catalogued in Thomson Reuter's Journal Citation Reports that indicates the frequency with which the "average article" in a specific journal has been cited in a particular year or period . Funding reported was defined as funding mentioned and provided for the conduct of the study. Journal adoption of the CONSORT statement at the time of data abstraction was defined as the article’s journal endorsing, encouraging or enforcing the CONSORT statement for study submissions at the time the article was reviewed for this study.
Journal impact factor, sample size and declared funding have been correlated with better reporting [3, 7, 25, 44–48]. Journal adoption of the CONSORT statement has also been associated with improved reporting of RCTs [6, 7, 10, 11, 22]. We hypothesized that studies with larger sample sizes, that have funding reported, that are published in journals with a higher impact factor and that are in journals that have adopted, endorsed or supported the CONSORT statement will be associated with better quality of reporting and reporting of key methodological items.
The statistical analysis was modeled based on the analysis done in a previous study looking at the quality of reporting of general endocrinology literature . The predictor variables, journal impact factor and sample size were also assessed in this study . The new predictor variable included for our study is journal adoption, endorsement or support of the CONSORT statement at the time of data abstraction and funding reported.
Categorical data were reported as the number of counts. The number of trials that clearly stated the element and the associated 95% confidence interval (CI) were calculated. For the element that had “zero” count or “full” counts (i.e. when none of or all of the included trials reported that element respectively), the 95% CI was calculated by adopting the rule of three . If none of the n studies showed the element that we were interested in, we could be 95% confident that the chance of this event occurring is at most 3 in n . For the other elements, the 95% CI of the count was calculated by assuming that the number of trials that clearly stated the element followed a binomial distribution. The probability that a trial had clearly stated the element was set to be the observed probability in the sample. The Cohen’s Kappa (κ) statistics were used to calculate the chance-adjusted agreement between the two raters for the 15 quality of reporting items, the 2 additional items (ethical issues and flow chart) and the 3 key methodological items. Agreement was interpreted as poor if κ ≤ 0.2; fair if 0.21 ≤ κ ≤ 0.4; moderate if 0.41 ≤ κ ≤0.6; substantial if 0.61 ≤ κ ≤ 0.8; and good if κ > 0.8 .
A multivariable regression analysis with the OQRS as an outcome variable was conducted to determine if a higher sample size, a higher impact factor, the presence of funding reported and the journal adoption, endorsement or support of the CONSORT statement was independently associated with a better OQRS. These four variables were used as predictors. The sample size was transformed by the logarithm function with base 10 to improve interpretability. The OQRS (discrete, ranged from 0–15) was assumed to follow a Poisson distribution. Incidence rate ratios (IRR) were used to express the results of the analysis. Variables were considered to be statistically significant at alpha (α) = 0.05. The same approach was followed for regression analysis with the KMIS as the outcome variable. It is also important to note that we used Variance Inflation Factors (VIFs) to assess collinearity between predictors and none of the VIFs were greater than 10 showing no major collinearity among the variables. SPSS© 220.127.116.11 (IBM Corporation, 2010) was used to perform the statistical analysis.
Rating of quality of reporting
Overall Quality of Reporting: Rating using items from the CONSORT Statement and two additional items
All Articles (n = 23)
1. Title or Abstract
7. Sample Size
8. Randomization: Sequence Generation
9. Randomization: Implementation
10. Statistical Methods
11. Participants Flow
13. Baseline Data
14. Outcomes and Estimates
a. Ethical Issues
Rating of key methodological items
Reporting Quality of Key Methodology Items
All Articles (n = 23)
Allocation of Concealment
Allocation was considered appropriate in a RCT study if one of the following allocation methods were reported; 1) central randomization, 2) numbered, coded vehicles, and 3) opaque, sealed, and sequentially numbered envelopes
Blinding was considered to have occurred if at least one specific group was explicitly reported as blinded if there was a blinding feasibility issue. In cases where blinding was not an issue, at least two groups must have been explicitly reported as blinding to qualify as appropriate blinding.
Numbers Analysed (Intention to treat)
Intention to treat was defined as the inclusion of all patients randomly assigned in the analysis, regardless of whether they actually satisfied the entry criteria, the treatment actually received, and subsequent with withdrawal or protocol deviations.
An appropriate method of allocation concealment was reported in only 8 of the 23 RCTs (34.8%), with opaque, sealed and sequentially numbered envelopes being the only method reported by all of these 8 RCTs. Most reports, 15 of 23 (65.2%) did not give any information about this key methodological item.
Only 6 of the 23 RCTs (26.1%) were blinded according to the study definition. Among the 18 studies where blinding of patients was considered feasible, only 4 of those 18 studies specifically reported at least two blinded groups. Among the 5 studies where blinding of patients was considered not feasible, 2 reported blinding of another group. Table 3 shows the number of studies reporting key methodological items.
Factors associated with reporting quality
Regression analysis for the Overall Reporting Quality a in 23 RCTs
Journal adopted CONSORT Statement at the time of data collection.
Regression analysis for the Key Methodology Item Score a in 23 RCTs
Journal adopted CONSORT Statement at the time of data collection.
In this study we assessed the RCT reporting quality of 23 RCTs used within a FNB meta-analysis . Most of these articles came from anesthesia specialty journals, except for 3 of the RCTs. All RCTs were related to post-operative management of analgesia after total knee arthroplasty. The overall quality of reporting of the RCTs assessed was poor to moderate, with only 4 of the 15 items being reported in over 90% of the articles. We have identified several areas where reporting of a particular item was insufficient in most of the studies assessed. These areas included: Item 1) Title and Abstract, Item 4) Participants, Item 6) Outcomes, Item 8) Randomization: Sequence Generation, Item 9) Randomization: Implementation, Item 11) Participants Flow, Item 12) Recruitment and Item 14) Outcomes and Estimation. Areas where items were reported moderately well were: Item 7) Sample size, Item 10) Statistical Methods and Item 15) Harms. Also very noteworthy was that the reporting of the key methodological items was poor with less than 50% of the articles reporting any of the 3 key methodological items (Table 3). Our results agree with many studies which have assessed the quality of reporting of RCTs published in medical and subspecialty journals, including studies which have focused on journals specialized in anesthesia [3, 17, 21, 23–29, 44–46]. All of these studies showed there is poor quality of reporting, with the 3 key methodological items assessed in this study (appropriate allocation concealment mechanism, blinding and numbers analysed by ITT) being highlighted as poorly reported [17, 21, 23–27, 35, 44]. Even though there is evidence to suggest that the quality of reporting has significantly improved over time, especially with the onset, use and publication of the CONSORT statement, the poor quality of reporting overall suggests that the statistically significant improvements are not enough to be clinically important as higher quality of reporting is needed to reduce bias and properly support clinicians’ decisions about treatment management [13, 20, 22, 35]. Also, the overall quality of reporting does not necessarily correspond with the proper reporting of the 3 key methodological items assessed for in this study; 3 items which are deemed important in preventing bias [9, 35]. Allocation concealment is important in avoiding selection bias, proper blinding is important in avoiding performance and detection bias, and numbers analyzed (ITT principle) is important in avoiding attrition bias . Any method of randomization, allocation concealment or blinding which allows the investigator to control the treatment group to which the study participant will be assigned should be avoided . Recent studies suggest that deficiency and poor quality of reporting of these 3 key methodological items often causes bias in the estimate of treatment effects, though the extent of bias can be unpredictable [27, 31, 32]. The overall poor quality of reporting in anesthesia literature is important to highlight as it suggests that there is a need to standardize the reporting of RCTs in anesthesia publications where information is retrieved, disseminated, and used for clinical practice. Although the CONSORT group was created to help improve reporting, there seems to be a need to register the urgency for completeness, clarity and transparency of reporting  in order to make publications more accessible and easily comparable with one another. This is apparent with the 23 RCTs assessed for this study as well since all of these articles were published after the CONSORT statement was established in 1996. The introduction of the CONSORT among journals and enforcing its use by requiring authors to submit a CONSORT checklist has the potential to greatly improve the quality of reporting for its readers and researchers needing to use the important information that RCTs provide for future application and meta-analysis. This notion is supported by evidence shown in 3 recent observational studies demonstrating that the introduction of the CONSORT statement is associated with an improvement in the quality of reporting [20, 22, 51].
There are many limitations to this study. One of the key limitations is the small sample size of RCTs used to evaluate the quality of reporting and key methodological items. The reason for this is that our intention was to evaluate the reporting quality of RCTs used in an actual meta-analysis. In hindsight, our study may not have had enough power to detect any significant association with our four predictor variables. In order to compensate for this we looked at trends among our findings with two other studies [3, 25] that looked at analogous predictor variables using comparable methods of statistical analysis, and their association with the overall quality of reporting or reporting of key methodological items in RCTs. We found that two of the predictor variables we looked at (Sample Size and Impact Factor) had shown a significant association with reporting quality in these two studies. We were not able to compare our other two predictor variables (Journal adoption of CONSORT statement at the time of data collection and funding reported) with these two comparator studies and with other studies that were reviewed because of one of three reasons: 1) the variable was not used as a predictor variable, 2) the findings were not reported in a numerical and statistical manner that could be used for trend comparison, or 3) it was not reported at all in publication. For the OQRS there were a few noted trends. Though the direction and the magnitude of the effect of sample size was not seen as similar to the two comparator studies, our 95% CI range for sample size encompassed the IRR of both comparator studies suggesting that there is a possibility that our results would have showed association had the sample size been larger. However, for the predictor variable of impact factor, which was measured in our study on a continuous scale, there were no trends observed in the one study we could use as a comparison. It should be noted that the comparator paper we could use for impact factor measured impact factor association in a categorical manner. Most of the impact factors of the RCT journals in our study fell into only one of those categories suggesting that our CI range may have encompassed their model had we been able to compare on a continuous scale. We also had 4 RCTs that came from journals with no impact factor evaluation for 2010. This decreased our RCT sample size from 23 to 19, further reducing our power to detect a difference.
For the KMIS there were also a few noted trends with one of the comparator studies. Both the sample size and impact factor of our results matched the direction and range found in the study by Lai et al. suggesting that our results would have reached significance had the sample size been larger. This trend is also more likely to be apparent with KMIS, even with the sample size we had, because of noted studies suggesting that key methodological items, such as the three used in this study, are associated with improved quality of reporting [3, 7, 25, 44–46] and the importance the 3 key methodological items assessed (Blinding, Allocation Concealment and Numbers Analysed) have in reducing bias [6, 31, 32, 36, 52]. In the future it might be of interest to assess the quality of RCTs from more than one meta-analysis on a related topic or even a random sample of RCTs in recent years in primary journals. Another limitation of our study is the fact that we cannot verify the degree with which the quality of reporting reflects the true methodological quality of the RCTs we assessed. The lack of reporting of a particular item in an RCT does not mean that the study in truth had a poor methodology. It is possible that the protocols of many RCTs include important aspects of methodology however important methodological details were left out of the published report [53, 54]. In one study it was reported that the proportion of double blinded trials with a clear description of the blinding of participants increased from 19%, with reliance on the publication alone, to 67% when the protocol for the RCT was reviewed and supplemented for assessment with the publication . Although the reporting on the key methodological item of blinding was found to be inadequate in both trial protocols and publications, the results show that there is still a possibility that the methodological and reporting quality of an RCT is greater than what is assessed in the published report [55, 56]. Soares et al. also showed that adequate concealment of allocation was achieved in all protocols but was only reported in 42% of the published papers . Reviewing protocols may improve the quality of reporting assessment however the published report is an important proxy in determining the validity and clinically relevant effect estimate as it is the source most accessible to clinicians and researchers. The quality of the report itself is important for this purpose. An additional limitation was the fact there is no standard instrument to evaluate the quality of reporting of RCTs. The majority of scales have not been thoroughly developed or tested for in regards to validity and reliability as a gold standard (criterion validity) is needed to compare it against. A gold standard does not exist for assessing the quality of RCTs and so checklists demarcating important features of quality and scales based on a quality of reporting checklist are assessed based on face validity and content validity from a theoretical model founded on accepted criteria [57, 58]. Quality scores based on checklists are also unreliable and may introduce bias as scores were found to differ depending on the scoring system that was used [59, 60]. Our results may also not entirely represent the quality of reporting of RCTs used in meta-analyses related to the intervention of the meta-analysis we used . Therefore the assessment of the quality of reporting of the RCTs used in other anesthesia related meta-analyses may help in compiling a sufficient pool of studies providing a sample size with enough power to detect a significant difference. Even with our limitations our results have good internal validity as we had two qualified reviewers and good inter-rater correlation despite that there was a lack of clarity with the reporting of evaluated items. In the future, one important item to add to the OQRS is trial registration. Trial registration is another recently enforced part of any published RCT article in order to ensure accessibility of all research. Studies have shown that the presence of registration with a trial registry is suboptimal [61, 62] and trial registration is associated with improved reporting in RCTs within the highest ranked journals . It is also important to note that the year of publication definitely plays a role in the reporting quality of the RCTs chosen for our study. The CONSORT statement was established in 1996 and the speed of implementation of the guideline from the CONSORT group has been substandard as there continue to be observational studies concluding inadequate quality of reporting of many RCTs dated from 1996 till now [1, 17, 21, 23–25]. Indeed, it has been shown that the total quality of reporting score is significantly associated with factors such as trial size, publication year, and the impact factor which the RCT is published within [44, 47, 48]. With that said the medical research community should also take notice that impact factor has been shown to be statistically significant as a predictor for better reporting quality on several occasions [25, 47, 48] and may be a feature to look for when wanting to find RCT publications that are of high reporting quality. It is possible that a large proportion of journals that have a higher impact factor have made initiatives to have authors submitting RCT manuscripts to their journal to adhere to the CONSORT statement. This may be worth investigating in the future to see if such a relationship exists.
In conclusion our study showed that the quality of reporting and reporting of key methodological items within anesthesia RCTs used for a post-operative pain management meta-analysis is poor to moderate. The knowledge gained from this study should be taken as another opportunity for journals to register the urgency for RCT publications to be complete, clear and transparent in reporting in order to make the literature accessible and comparable. Enforcing the use of the CONSORT statement by requiring authors to submit a CONSORT checklist can help greatly to improve the quality of reporting and get authors in the frame of reference needed to properly report studies or even structure study design.
Consolidated Standards for Reporting Trials
Femoral Nerve Block
Incidence Rate Ratio
Key Methodological Item Score
Randomized Controlled Trial
Overall Quality of Reporting
Statistical Package for the Social Sciences.
LT is the clinical trials mentor for the Canadian Institutes of Health Research. The study was funded in part by the CANNeCTIN programme and by Drug Safety and Effectiveness Cross-Disciplinary Training (DSECT) Program.
- Schulz KF, Altman DG, Moher D, CONSORT Group: CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMC Med. 2010, 8: 18-10.1186/1741-7015-8-18.PubMed CentralView ArticlePubMed
- Rios LP, Ye C, Thabane L: Association between framing of the research question using the PICOT format and reporting quality of randomized controlled trials. BMC Med Res Methodol. 2010, 10: 11-10.1186/1471-2288-10-11.PubMed CentralView ArticlePubMed
- Rios LP, Odueyungbo A, Moitri MO, Rahman MO, Thabane L: Quality of reporting of randomized controlled trials in general endocrinology literature. J Clin Endocrinol Metab. 2008, 93 (10): 3810-3816. 10.1210/jc.2008-0817.View ArticlePubMed
- Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995, 273 (5): 408-412. 10.1001/jama.1995.03520290060030.View ArticlePubMed
- Cochrane Handbook for Systematic Reviews of Intervention. , , , http://www.cochrane-handbook.org,
- Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, Elbourne D, Egger M, Altman DG: CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010, 340: c869-10.1136/bmj.c869.PubMed CentralView ArticlePubMed
- Can OS, Yilmaz AA, Hasdogan M, Alkaya F, Turhan SC, Can MF, Alanoglu Z: Has the quality of abstracts for randomised controlled trials improved since the release of Consolidated Standards of Reporting Trial guideline for abstract reporting? A survey of four high-profile anaesthesia journals. Eur J Anaesthesiol. 2011, 28 (7): 485-492. 10.1097/EJA.0b013e32833fb96f.View ArticlePubMed
- Higgins JPT, Altman DG, Sterne JAC: Chapter 8 Assessing risk of bias in included studies. Cochrane Handbook for Systematic Reviews of Interventions. Edited by: Higgins JPT, Green S. 2011, The Cochrane Collaboration, , 8.0-510
- Juni P, Altman DG, Egger M: Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ. 2001, 323 (7303): 42-46. 10.1136/bmj.323.7303.42.PubMed CentralView ArticlePubMed
- Moher D, Jones A, Lepage L, CONSORT Group (Consolitdated Standards for Reporting of Trials): Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA. 2001, 285 (15): 1992-1995. 10.1001/jama.285.15.1992.View ArticlePubMed
- Moher D, Schulz KF, Altman DG: The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001, 357 (9263): 1191-1194. 10.1016/S0140-6736(00)04337-3.View ArticlePubMed
- Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP: Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?. Lancet. 1998, 352 (9128): 609-613. 10.1016/S0140-6736(98)01085-X.View ArticlePubMed
- Moher D, Altman DG, Schulz KF, Elbourne DR: Opportunities and challenges for improving the quality of reporting clinical research: CONSORT and beyond. CMAJ. 2004, 171 (4): 349-350. 10.1503/cmaj.1040031.PubMed CentralView ArticlePubMed
- Higgins JPT, Green S: Cochrane Collaboration: Cochrane handbook for systematic reviews of interventions. 2008, Wiley-Blackwell, Chichester, West Sussex; Hoboken NJView Article
- Balk EM, Bonis PA, Moskowitz H, Schmid CH, Ioannidis JP, Wang C, Lau J: Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA. 2002, 287 (22): 2973-2982. 10.1001/jama.287.22.2973.View ArticlePubMed
- Kjaergard LL, Villumsen J, Gluud C: Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med. 2001, 135 (11): 982-989.View ArticlePubMed
- Chan AW, Altman DG: Epidemiology and reporting of randomised trials published in PubMed journals. Lancet. 2005, 365 (9465): 1159-1162. 10.1016/S0140-6736(05)71879-1.View ArticlePubMed
- Glasziou P, Meats E, Heneghan C, Shepperd S: What is missing from descriptions of treatment in trials and reviews?. BMJ. 2008, 336 (7659): 1472-1474. 10.1136/bmj.39590.732037.47.PubMed CentralView ArticlePubMed
- Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, Decullier E, Easterbrook PJ, Von Elm E, Gamble C, Ghersi D, Ioannidis JP, Simes J, Williamson PR: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008, 3 (8): e3081-10.1371/journal.pone.0003081.PubMed CentralView ArticlePubMed
- Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG: The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ. 2010, 340: c723-10.1136/bmj.c723.PubMed CentralView ArticlePubMed
- Mills EJ, Wu P, Gagnier J, Devereaux PJ: The quality of randomized trial reporting in leading medical journals since the revised CONSORT statement. Contemp Clin Trials. 2005, 26 (4): 480-487. 10.1016/j.cct.2005.02.008.View ArticlePubMed
- Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, Gaboury I: Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust. 2006, 185 (5): 263-267.PubMed
- Lai TY, Wong VW, Lam RF, Cheng AC, Lam DS, Leung GM: Quality of reporting of key methodological items of randomized controlled trials in clinical ophthalmic journals. Ophthalmic Epidemiol. 2007, 14 (6): 390-398. 10.1080/09286580701344399.View ArticlePubMed
- Mills E, Loke YK, Wu P, Montori VM, Perri D, Moher D, Guyatt G: Determining the reporting quality of RCTs in clinical pharmacology. Br J Clin Pharmacol. 2004, 58 (1): 61-65. 10.1111/j.1365-2125.2004.2092.x.PubMed CentralView ArticlePubMed
- Lai R, Chu R, Fraumeni M, Thabane L: Quality of randomized controlled trials reporting in the primary treatment of brain tumors. J Clin Oncol. 2006, 24 (7): 1136-1144. 10.1200/JCO.2005.03.1179.View ArticlePubMed
- Greenfield ML, Mhyre JM, Mashour GA, Blum JM, Yen EC, Rosenberg AL: Improvement in the quality of randomized controlled trials among general anesthesiology journals 2000 to 2006: a 6-year follow-up. Anesth Analg. 2009, 108 (6): 1916-1921. 10.1213/ane.0b013e31819fe6d7.View ArticlePubMed
- Greenfield ML, Rosenberg AL, O'Reilly M, Shanks AM, Sliwinski MJ, Nauss MD: The quality of randomized controlled trials in major anesthesiology journals. Anesth Analg. 2005, 100 (6): 1759-1764. 10.1213/01.ANE.0000150612.71007.A3.View ArticlePubMed
- Pua HL, Lerman J, Crawford MW, Wright JG: An evaluation of the quality of clinical trials in anesthesia. Anesthesiology. 2001, 95 (5): 1068-1073. 10.1097/00000542-200111000-00007.View ArticlePubMed
- Paul JE, Arya A, Hurlburt L, Cheng J, Thabane L, Tidy A, Murthy Y: Femoral nerve block improves analgesia outcomes after total knee arthroplasty: a meta-analysis of randomized controlled trials. Anesthesiology. 2010, 113 (5): 1144-1162. 10.1097/ALN.0b013e3181f4b18.View ArticlePubMed
- Chung KC, Kotsis SV: The ethics of clinical research. J Hand Surg Am. 2011, 36 (2): 308-315. 10.1016/j.jhsa.2010.11.012.View ArticlePubMed
- Nuesch E, Reichenbach S, Trelle S, Rutjes AW, Liewald K, Sterchi R, Altman DG, Juni P: The importance of allocation concealment and patient blinding in osteoarthritis trials: a meta-epidemiologic study. Arthritis Rheum. 2009, 61 (12): 1633-1641. 10.1002/art.24894.View ArticlePubMed
- Nuesch E, Trelle S, Reichenbach S, Rutjes AW, Burgi E, Scherer M, Altman DG, Juni P: The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study. BMJ. 2009, 339: b3244-10.1136/bmj.b3244.PubMed CentralView ArticlePubMed
- Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, Gluud C, Martin RM, Wood AJ, Sterne JA: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008, 336 (7644): 601-605. 10.1136/bmj.39465.451748.AD.PubMed CentralView ArticlePubMed
- Schulz KF, Grimes DA, Altman DG, Hayes RJ: Blinding and exclusions after allocation in randomised controlled trials: survey of published parallel group trials in obstetrics and gynaecology. BMJ. 1996, 312 (7033): 742-744. 10.1136/bmj.312.7033.742.PubMed CentralView ArticlePubMed
- Huwiler-Muntener K, Juni P, Junker C, Egger M: Quality of reporting of randomized trials as a measure of methodologic quality. JAMA. 2002, 287 (21): 2801-2804. 10.1001/jama.287.21.2801.View ArticlePubMed
- Montori VM, Bhandari M, Devereaux PJ, Manns BJ, Ghali WA, Guyatt GH: In the dark: the reporting of blinding status in randomized controlled trials. J Clin Epidemiol. 2002, 55 (8): 787-790. 10.1016/S0895-4356(02)00446-8.View ArticlePubMed
- Haahr MT, Hrobjartsson A: Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors. Clin Trials. 2006, 3 (4): 360-365.PubMed
- Miller LE, Stewart ME: The blind leading the blind: use and misuse of blinding in randomized controlled trials. Contemp Clin Trials. 2011, 32 (2): 240-243. 10.1016/j.cct.2010.11.004.View ArticlePubMed
- Newell DJ: Intention-to-treat analysis: implications for quantitative and qualitative research. Int J Epidemiol. 1992, 21 (5): 837-841. 10.1093/ije/21.5.837.View ArticlePubMed
- Lewis JA, Machin D: Intention to treat–who should use ITT?. Br J Cancer. 1993, 68 (4): 647-650. 10.1038/bjc.1993.402.PubMed CentralView ArticlePubMed
- Higgins JPT, Deeks JJ, Altman DG: Chapter 16: Special Topics in statistics. Cochrane Handbook for Systematic Reviews and Interventions. Edited by: Higgins JPT, Green S. 2011, The Cochrane Collaboration,, 16.0-16.2. http://www.cochrane-handbook.org, 2011,
- Hollis S, Campbell F: What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ. 1999, 319 (7211): 670-674. 10.1136/bmj.319.7211.670.PubMed CentralView ArticlePubMed
- The Thomson Reuters Impact Factor. , , , http://thomsonreuters.com/products_services/science/free/essays/impact_factor,
- Farrokhyar F, Chu R, Whitlock R, Thabane L: A systematic review of the quality of publications reporting coronary artery bypass grafting trials. Can J Surg. 2007, 50 (4): 266-277.PubMed CentralPubMed
- Thomas O, Thabane L, Douketis J, Chu R, Westfall AO, Allison DB: Industry funding and the reporting quality of large long-term weight loss trials. Int J Obes (Lond). 2008, 32 (10): 1531-1536. 10.1038/ijo.2008.137.View Article
- Balasubramanian SP, Wiener M, Alshameeri Z, Tiruvoipati R, Elbourne D, Reed MW: Standards of reporting of randomized controlled trials in general surgery: can we do better?. Ann Surg. 2006, 244 (5): 663-667. 10.1097/01.sla.0000217640.11224.05.PubMed CentralView ArticlePubMed
- Toulmonde M, Bellera C, Mathoulin-Pelissier S, Debled M, Bui B, Italiano A: Quality of Randomized Controlled Trials Reporting in the Treatment of Sarcomas. J Clin Oncol. 2011, 29 (9): 1204-1209. 10.1200/JCO.2010.30.9369.View ArticlePubMed
- Ivers NM, Taljaard M, Dixon S, Bennett C, McRae A, Taleban J, Skea Z, Brehaut JC, Boruch RF, Eccles MP, Grimshaw JM, Weijer C, Zwarenstein M, Donner A: Impact of CONSORT extension for cluster randomised trials on quality of reporting and study methodology: review of random sample of 300 trials, 2000–8. BMJ. 2011, 343: d5886-10.1136/bmj.d5886.PubMed CentralView ArticlePubMed
- Eypasch E, Lefering R, Kum CK, Troidl H: Probability of adverse events that have not yet occurred: a statistical reminder. BMJ. 1995, 311 (7005): 619-620. 10.1136/bmj.311.7005.619.PubMed CentralView ArticlePubMed
- Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics. 1977, 33 (1): 159-174. 10.2307/2529310.View ArticlePubMed
- Egger M, Juni P, Bartlett C, CONSORT Group (Consolidated Standards of Reporting of Trials): Value of flow diagrams in reports of randomized controlled trials. JAMA. 2001, 285 (15): 1996-1999. 10.1001/jama.285.15.1996.View ArticlePubMed
- Montori VM, Guyatt GH: Intention-to-treat principle. CMAJ. 2001, 165 (10): 1339-1341.PubMed CentralPubMed
- Devereaux PJ, Choi PT, El-Dika S, Bhandari M, Montori VM, Schunemann HJ, Garg AX, Busse JW, Heels-Ansdell D, Ghali WA, Manns BJ, Guyatt GH: An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol. 2004, 57 (12): 1232-1236. 10.1016/j.jclinepi.2004.03.017.View ArticlePubMed
- Pildal J, Chan AW, Hrobjartsson A, Forfang E, Altman DG, Gotzsche PC: Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study. BMJ. 2005, 330 (7499): 1049-10.1136/bmj.38414.422650.8F.PubMed CentralView ArticlePubMed
- Hrobjartsson A, Pildal J, Chan AW, Haahr MT, Altman DG, Gotzsche PC: Reporting on blinding in trial protocols and corresponding publications was often inadequate but rarely contradictory. J Clin Epidemiol. 2009, 62 (9): 967-973. 10.1016/j.jclinepi.2009.04.003.View ArticlePubMed
- Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, Djulbegovic B: Radiation Therapy Oncology Group: Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ. 2004, 328 (7430): 22-24. 10.1136/bmj.328.7430.22.PubMed CentralView ArticlePubMed
- Verhagen AP, de Vet HC, de Bie RA, Boers M, van den Brandt PA: The art of quality assessment of RCTs included in systematic reviews. J Clin Epidemiol. 2001, 54 (7): 651-654. 10.1016/S0895-4356(00)00360-7.View ArticlePubMed
- Olivo SA, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ: Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther. 2008, 88 (2): 156-175. 10.2522/ptj.20070147.View ArticlePubMed
- Juni P, Witschi A, Bloch R, Egger M: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999, 282 (11): 1054-1060. 10.1001/jama.282.11.1054.View ArticlePubMed
- Herbison P, Hay-Smith J, Gillespie WJ: Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol. 2006, 59 (12): 1249-1256.View ArticlePubMed
- Agha R, Cooper D, Muir G: The reporting quality of randomised controlled trials in surgery: A systematic review. Int J Surg. 2007, 5 (6): 413-422. 10.1016/j.ijsu.2007.06.002.View ArticlePubMed
- Reveiz L, Cortés-Jofré M, Asenjo Lobos C, Nicita G, Ciapponi A, Garcìa-Dieguez M, Tellez D, Delgado M, Solà I, Ospina E: Influence of trial registration on reporting quality of randomized trials: Study from highest ranked journals. J Clin Epidemiol. 2010, 63 (11): 1216-1222. 10.1016/j.jclinepi.2010.01.013.View ArticlePubMed
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2253/12/13/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.