Assessing risk of bias in non-randomised studies and incorporating GRADE: Initial experience with a new Cochrane ‘risk of bias’ tool under development

Steven MacLennan, Mari Imamura, P Dahm, Molly Nueberger, B. C. Reeves, Graeme Stewart MacLennan, Muhammad Imran Omar, Sam McClinton, T R Leyshon Griffiths, James Michael Olu N'Dow

Research output: Contribution to conferencePaper

Abstract

Background:In instances where randomised controlled trials (RCT) are impossible or have not been conducted, clinical recommendations and decision-making must rely on other evidence. If systematic reviewers decide to include non-randomised studies (NRS), it is imperative to use a standard method to assess and communicate the risk of bias (RoB) in NRS.

Objectives:To pilot a RoB tool for NRS and make it commensurate with GRADE.

Methods:An extended version of the Cochrane RCT RoB tool was applied to NRS. This included an additional item on the risk of findings of an NRS being explained by confounding. Each pre-specified confounding factor was assessed on the precision of measurement, baseline imbalance, and quality of case-mix adjustment, on 5-point scales. Imbalance was judged by clinical consensus, while other items were assessed by two independent reviewers. Mean ‘adjustment’ scores per outcome across studies were used to determine the GRADE quality of evidence. The tool was applied to 33 NRS retrieved for a systematic review of surgical interventions for localised renal cancer.

Results:The initial 5-point scale was unwieldy and lead to disagreement among reviewers. We created scoring guidelines and re-piloted. RoB scores were tabulated rather than aggregated to indicate where likely biases were located. All NRS were rated as either ‘low’ or ‘very low on GRADE; however, determining an appropriate cut-off required considerable judgement.

Conclusions:Compared with RoB assessment in RCT, assessment of NRS was more difficult and increased required time and expertise resources. In areas where the quality of studies is known to be very low, the added time and complexity may make the assessment not worthwhile. Presentation of the large amount of information generated by this tool is challenging. Further research needs to strike a balance of making a ‘brief’ and ‘easy’ version while addressing complex methodological issues inherent
Original languageEnglish
Publication statusPublished - Oct 2011
EventCochrane Colloquium 2011 - Madrid, Spain
Duration: 18 Oct 201122 Oct 2011

Conference

ConferenceCochrane Colloquium 2011
CountrySpain
CityMadrid
Period18/10/1122/10/11

Fingerprint

Randomized Controlled Trials
Risk Adjustment
Social Adjustment
Kidney Neoplasms
Consensus
Outcome Assessment (Health Care)
Guidelines
Research
Non-Randomized Controlled Trials
Clinical Decision-Making

Cite this

Assessing risk of bias in non-randomised studies and incorporating GRADE : Initial experience with a new Cochrane ‘risk of bias’ tool under development. / MacLennan, Steven; Imamura, Mari; Dahm, P; Nueberger, Molly; Reeves, B. C.; MacLennan, Graeme Stewart; Omar, Muhammad Imran; McClinton, Sam; Griffiths, T R Leyshon; N'Dow, James Michael Olu.

2011. Paper presented at Cochrane Colloquium 2011, Madrid, Spain.

Research output: Contribution to conferencePaper

@conference{cccc204ea9b54febbf4dcfdee7aa92d7,
title = "Assessing risk of bias in non-randomised studies and incorporating GRADE: Initial experience with a new Cochrane ‘risk of bias’ tool under development",
abstract = "Background:In instances where randomised controlled trials (RCT) are impossible or have not been conducted, clinical recommendations and decision-making must rely on other evidence. If systematic reviewers decide to include non-randomised studies (NRS), it is imperative to use a standard method to assess and communicate the risk of bias (RoB) in NRS. Objectives:To pilot a RoB tool for NRS and make it commensurate with GRADE. Methods:An extended version of the Cochrane RCT RoB tool was applied to NRS. This included an additional item on the risk of findings of an NRS being explained by confounding. Each pre-specified confounding factor was assessed on the precision of measurement, baseline imbalance, and quality of case-mix adjustment, on 5-point scales. Imbalance was judged by clinical consensus, while other items were assessed by two independent reviewers. Mean ‘adjustment’ scores per outcome across studies were used to determine the GRADE quality of evidence. The tool was applied to 33 NRS retrieved for a systematic review of surgical interventions for localised renal cancer. Results:The initial 5-point scale was unwieldy and lead to disagreement among reviewers. We created scoring guidelines and re-piloted. RoB scores were tabulated rather than aggregated to indicate where likely biases were located. All NRS were rated as either ‘low’ or ‘very low on GRADE; however, determining an appropriate cut-off required considerable judgement. Conclusions:Compared with RoB assessment in RCT, assessment of NRS was more difficult and increased required time and expertise resources. In areas where the quality of studies is known to be very low, the added time and complexity may make the assessment not worthwhile. Presentation of the large amount of information generated by this tool is challenging. Further research needs to strike a balance of making a ‘brief’ and ‘easy’ version while addressing complex methodological issues inherent",
author = "Steven MacLennan and Mari Imamura and P Dahm and Molly Nueberger and Reeves, {B. C.} and MacLennan, {Graeme Stewart} and Omar, {Muhammad Imran} and Sam McClinton and Griffiths, {T R Leyshon} and N'Dow, {James Michael Olu}",
year = "2011",
month = "10",
language = "English",
note = "Cochrane Colloquium 2011 ; Conference date: 18-10-2011 Through 22-10-2011",

}

TY - CONF

T1 - Assessing risk of bias in non-randomised studies and incorporating GRADE

T2 - Initial experience with a new Cochrane ‘risk of bias’ tool under development

AU - MacLennan, Steven

AU - Imamura, Mari

AU - Dahm, P

AU - Nueberger, Molly

AU - Reeves, B. C.

AU - MacLennan, Graeme Stewart

AU - Omar, Muhammad Imran

AU - McClinton, Sam

AU - Griffiths, T R Leyshon

AU - N'Dow, James Michael Olu

PY - 2011/10

Y1 - 2011/10

N2 - Background:In instances where randomised controlled trials (RCT) are impossible or have not been conducted, clinical recommendations and decision-making must rely on other evidence. If systematic reviewers decide to include non-randomised studies (NRS), it is imperative to use a standard method to assess and communicate the risk of bias (RoB) in NRS. Objectives:To pilot a RoB tool for NRS and make it commensurate with GRADE. Methods:An extended version of the Cochrane RCT RoB tool was applied to NRS. This included an additional item on the risk of findings of an NRS being explained by confounding. Each pre-specified confounding factor was assessed on the precision of measurement, baseline imbalance, and quality of case-mix adjustment, on 5-point scales. Imbalance was judged by clinical consensus, while other items were assessed by two independent reviewers. Mean ‘adjustment’ scores per outcome across studies were used to determine the GRADE quality of evidence. The tool was applied to 33 NRS retrieved for a systematic review of surgical interventions for localised renal cancer. Results:The initial 5-point scale was unwieldy and lead to disagreement among reviewers. We created scoring guidelines and re-piloted. RoB scores were tabulated rather than aggregated to indicate where likely biases were located. All NRS were rated as either ‘low’ or ‘very low on GRADE; however, determining an appropriate cut-off required considerable judgement. Conclusions:Compared with RoB assessment in RCT, assessment of NRS was more difficult and increased required time and expertise resources. In areas where the quality of studies is known to be very low, the added time and complexity may make the assessment not worthwhile. Presentation of the large amount of information generated by this tool is challenging. Further research needs to strike a balance of making a ‘brief’ and ‘easy’ version while addressing complex methodological issues inherent

AB - Background:In instances where randomised controlled trials (RCT) are impossible or have not been conducted, clinical recommendations and decision-making must rely on other evidence. If systematic reviewers decide to include non-randomised studies (NRS), it is imperative to use a standard method to assess and communicate the risk of bias (RoB) in NRS. Objectives:To pilot a RoB tool for NRS and make it commensurate with GRADE. Methods:An extended version of the Cochrane RCT RoB tool was applied to NRS. This included an additional item on the risk of findings of an NRS being explained by confounding. Each pre-specified confounding factor was assessed on the precision of measurement, baseline imbalance, and quality of case-mix adjustment, on 5-point scales. Imbalance was judged by clinical consensus, while other items were assessed by two independent reviewers. Mean ‘adjustment’ scores per outcome across studies were used to determine the GRADE quality of evidence. The tool was applied to 33 NRS retrieved for a systematic review of surgical interventions for localised renal cancer. Results:The initial 5-point scale was unwieldy and lead to disagreement among reviewers. We created scoring guidelines and re-piloted. RoB scores were tabulated rather than aggregated to indicate where likely biases were located. All NRS were rated as either ‘low’ or ‘very low on GRADE; however, determining an appropriate cut-off required considerable judgement. Conclusions:Compared with RoB assessment in RCT, assessment of NRS was more difficult and increased required time and expertise resources. In areas where the quality of studies is known to be very low, the added time and complexity may make the assessment not worthwhile. Presentation of the large amount of information generated by this tool is challenging. Further research needs to strike a balance of making a ‘brief’ and ‘easy’ version while addressing complex methodological issues inherent

M3 - Paper

ER -