Abstract
Background
Systematic reviews of randomised trials guide policy and healthcare decisions. Yet, we observed that some reviews judge randomised trials as high or unclear risk of bias (ROB) for sequence generation, potentially introducing bias. However, to date, the extent of this issue has not been well examined. We evaluated the consistency in the ROB assessment for sequence generation of randomised trials in Cochrane and non-Cochrane reviews, and explored the reviewers’ judgement of the quality of evidence for the related outcomes.
Methods
Cochrane intervention reviews (01/01/2017–31/03/2017) were retrieved from the Cochrane Database of Systematic Reviews. We also searched for systematic reviews in ten general medical journals with highest impact factors (01/01/2016–31/03/2017). We examined the proportion of reviews that rated the sequence generation domain as high, low or unclear risk of selection bias. For reviews that had rated any randomised trials as high or unclear risk of bias, we examined the proportion that had assessed the quality of evidence.
Results
Overall, 100 systematic reviews were included in our analysis. We evaluated 64 Cochrane reviews which comprised of 984 randomised trials; 0.8% (n = 8) and 52.2% (n = 514) were rated as high and unclear ROB for sequence generation respectively. We further evaluated 36 non-Cochrane reviews which comprised of 1376 trials; 5.8% (n = 80) and 39.6% (n = 545) were rated as high and unclear ROB respectively. Ninety percent (n = 10) of non-Cochrane reviews which rated randomised trials as high ROB for sequence generation did not report an underlying reason. All Cochrane reviews assessed the quality of evidence (GRADE). For the non-Cochrane reviews, only just over half had assessed the quality of evidence.
Conclusion
Systematic reviews of interventions frequently rate randomised trials as high or unclear ROB for sequence generation. In general, Cochrane reviews were more transparent than non-Cochrane reviews in ROB and quality of evidence assessment. The scientific community should more strongly promote consistent ROB assessment for sequence generation to minimise selection bias and support transparent quality of evidence assessment. Consistency ensures that appropriate conclusions are drawn from the data.
Systematic reviews of randomised trials guide policy and healthcare decisions. Yet, we observed that some reviews judge randomised trials as high or unclear risk of bias (ROB) for sequence generation, potentially introducing bias. However, to date, the extent of this issue has not been well examined. We evaluated the consistency in the ROB assessment for sequence generation of randomised trials in Cochrane and non-Cochrane reviews, and explored the reviewers’ judgement of the quality of evidence for the related outcomes.
Methods
Cochrane intervention reviews (01/01/2017–31/03/2017) were retrieved from the Cochrane Database of Systematic Reviews. We also searched for systematic reviews in ten general medical journals with highest impact factors (01/01/2016–31/03/2017). We examined the proportion of reviews that rated the sequence generation domain as high, low or unclear risk of selection bias. For reviews that had rated any randomised trials as high or unclear risk of bias, we examined the proportion that had assessed the quality of evidence.
Results
Overall, 100 systematic reviews were included in our analysis. We evaluated 64 Cochrane reviews which comprised of 984 randomised trials; 0.8% (n = 8) and 52.2% (n = 514) were rated as high and unclear ROB for sequence generation respectively. We further evaluated 36 non-Cochrane reviews which comprised of 1376 trials; 5.8% (n = 80) and 39.6% (n = 545) were rated as high and unclear ROB respectively. Ninety percent (n = 10) of non-Cochrane reviews which rated randomised trials as high ROB for sequence generation did not report an underlying reason. All Cochrane reviews assessed the quality of evidence (GRADE). For the non-Cochrane reviews, only just over half had assessed the quality of evidence.
Conclusion
Systematic reviews of interventions frequently rate randomised trials as high or unclear ROB for sequence generation. In general, Cochrane reviews were more transparent than non-Cochrane reviews in ROB and quality of evidence assessment. The scientific community should more strongly promote consistent ROB assessment for sequence generation to minimise selection bias and support transparent quality of evidence assessment. Consistency ensures that appropriate conclusions are drawn from the data.
| Original language | English |
|---|---|
| Article number | 13 |
| Journal | Systematic reviews |
| Volume | 8 |
| DOIs | |
| Publication status | Published - 8 Jan 2019 |
Bibliographical note
MR was supported by a Health Research Board (Ireland)—Trials Methodology Research Network Summer Student Scholarship.Keywords
- randomisation
- sequence generation
- systematic reviews
- risk of bias
- quality of evidence
- Risk of bias
- Quality of evidence
- Sequence generation
- Systematic reviews
- Randomisation