Initial experience with a pilot Cochrane tool for assessing risk of bias for non-randomized studies applying a web-based survey of content experts to derive criteria for imbalance

D. Palmer, Steven MacLennan, Mari Imamura, B. C. Reeves, A Clubb, A Duboy, Molly Nueberger, P Dahm

Research output: Contribution to conferencePaper

Abstract

Background: To date, few tools exist for assessing the risk of bias (ROB) from confounding variables in non-randomized studies (NRS).

Objectives: We tested a pilot Cochrane tool for ROB to assess its applicability and value in quantifying ROB in NRS.

Methods: We applied the Cochrane tool to a systematic review of NRS evaluating the effectiveness of lymph node dissection in patients undergoing radical cystectomy. Three independent review authors abstracted data on methodological quality using the Newcastle-Ottawa instrument and the Cochrane tool for assessing ROB.[1] The Cochrane tool assesses the precision, imbalance, and adjustment for each confounding variable, and overall ROB from confounding, blinding, incomplete outcome data and selective reporting, on 1 (low ROB) – 5 (high ROB) point scales. We pilot-tested the data abstraction form and developed detailed guidelines to optimize interobserver agreement.

Results: Five NRS met the inclusion criteria. Confounders were identified in advance by literature search and responses to a web-based survey of eleven bladder cancer experts who were also asked to categorize what difference in the proportion of a confounding variable represented “no”, a “small”, “moderate” or “large” imbalance. To evaluate imbalance, each confounder was evaluated dichotomously (e.g. organ-confined versus non-organ-confined). Threshold values were derived using the median responses; e.g. for non-organ-confined disease, these were set at = 2%, 2.1-5%, 5.1 – 7%, > 7%, respectively, thereby providing explicit anchors for imbalance criterion scoring.

Conclusions: The novel ROB tool represents a valuable instrument for quantifying the ROB of NRS that deserves further formal investigation in a larger, more diverse sample of studies. The described survey method appears an effective and efficient method to define the necessary imbalance criterion thresholds.

1. Reeves BC, Shea B, Wells GA. Classifying non-randomised studies (NRS) and the assessing the risk of bias for a systematic review. Workshop, 18th Cochrane Colloquium, Keystone 2010.
Original languageEnglish
Publication statusPublished - Oct 2011
EventCochrane Colloquium 2011 - Madrid, Spain
Duration: 18 Oct 201122 Oct 2011

Conference

ConferenceCochrane Colloquium 2011
CountrySpain
CityMadrid
Period18/10/1122/10/11

Fingerprint

Confounding Factors (Epidemiology)
Surveys and Questionnaires
Cystectomy
Lymph Node Excision
Urinary Bladder Neoplasms
Research Design
Guidelines
Education

Cite this

Initial experience with a pilot Cochrane tool for assessing risk of bias for non-randomized studies applying a web-based survey of content experts to derive criteria for imbalance. / Palmer, D.; MacLennan, Steven; Imamura, Mari; Reeves, B. C.; Clubb, A; Duboy, A; Nueberger, Molly; Dahm, P.

2011. Paper presented at Cochrane Colloquium 2011, Madrid, Spain.

Research output: Contribution to conferencePaper

Palmer, D, MacLennan, S, Imamura, M, Reeves, BC, Clubb, A, Duboy, A, Nueberger, M & Dahm, P 2011, 'Initial experience with a pilot Cochrane tool for assessing risk of bias for non-randomized studies applying a web-based survey of content experts to derive criteria for imbalance' Paper presented at Cochrane Colloquium 2011, Madrid, Spain, 18/10/11 - 22/10/11, .
@conference{6c420b120a9d4396aab048c563601022,
title = "Initial experience with a pilot Cochrane tool for assessing risk of bias for non-randomized studies applying a web-based survey of content experts to derive criteria for imbalance",
abstract = "Background: To date, few tools exist for assessing the risk of bias (ROB) from confounding variables in non-randomized studies (NRS). Objectives: We tested a pilot Cochrane tool for ROB to assess its applicability and value in quantifying ROB in NRS. Methods: We applied the Cochrane tool to a systematic review of NRS evaluating the effectiveness of lymph node dissection in patients undergoing radical cystectomy. Three independent review authors abstracted data on methodological quality using the Newcastle-Ottawa instrument and the Cochrane tool for assessing ROB.[1] The Cochrane tool assesses the precision, imbalance, and adjustment for each confounding variable, and overall ROB from confounding, blinding, incomplete outcome data and selective reporting, on 1 (low ROB) – 5 (high ROB) point scales. We pilot-tested the data abstraction form and developed detailed guidelines to optimize interobserver agreement. Results: Five NRS met the inclusion criteria. Confounders were identified in advance by literature search and responses to a web-based survey of eleven bladder cancer experts who were also asked to categorize what difference in the proportion of a confounding variable represented “no”, a “small”, “moderate” or “large” imbalance. To evaluate imbalance, each confounder was evaluated dichotomously (e.g. organ-confined versus non-organ-confined). Threshold values were derived using the median responses; e.g. for non-organ-confined disease, these were set at = 2{\%}, 2.1-5{\%}, 5.1 – 7{\%}, > 7{\%}, respectively, thereby providing explicit anchors for imbalance criterion scoring. Conclusions: The novel ROB tool represents a valuable instrument for quantifying the ROB of NRS that deserves further formal investigation in a larger, more diverse sample of studies. The described survey method appears an effective and efficient method to define the necessary imbalance criterion thresholds. 1. Reeves BC, Shea B, Wells GA. Classifying non-randomised studies (NRS) and the assessing the risk of bias for a systematic review. Workshop, 18th Cochrane Colloquium, Keystone 2010.",
author = "D. Palmer and Steven MacLennan and Mari Imamura and Reeves, {B. C.} and A Clubb and A Duboy and Molly Nueberger and P Dahm",
year = "2011",
month = "10",
language = "English",
note = "Cochrane Colloquium 2011 ; Conference date: 18-10-2011 Through 22-10-2011",

}

TY - CONF

T1 - Initial experience with a pilot Cochrane tool for assessing risk of bias for non-randomized studies applying a web-based survey of content experts to derive criteria for imbalance

AU - Palmer, D.

AU - MacLennan, Steven

AU - Imamura, Mari

AU - Reeves, B. C.

AU - Clubb, A

AU - Duboy, A

AU - Nueberger, Molly

AU - Dahm, P

PY - 2011/10

Y1 - 2011/10

N2 - Background: To date, few tools exist for assessing the risk of bias (ROB) from confounding variables in non-randomized studies (NRS). Objectives: We tested a pilot Cochrane tool for ROB to assess its applicability and value in quantifying ROB in NRS. Methods: We applied the Cochrane tool to a systematic review of NRS evaluating the effectiveness of lymph node dissection in patients undergoing radical cystectomy. Three independent review authors abstracted data on methodological quality using the Newcastle-Ottawa instrument and the Cochrane tool for assessing ROB.[1] The Cochrane tool assesses the precision, imbalance, and adjustment for each confounding variable, and overall ROB from confounding, blinding, incomplete outcome data and selective reporting, on 1 (low ROB) – 5 (high ROB) point scales. We pilot-tested the data abstraction form and developed detailed guidelines to optimize interobserver agreement. Results: Five NRS met the inclusion criteria. Confounders were identified in advance by literature search and responses to a web-based survey of eleven bladder cancer experts who were also asked to categorize what difference in the proportion of a confounding variable represented “no”, a “small”, “moderate” or “large” imbalance. To evaluate imbalance, each confounder was evaluated dichotomously (e.g. organ-confined versus non-organ-confined). Threshold values were derived using the median responses; e.g. for non-organ-confined disease, these were set at = 2%, 2.1-5%, 5.1 – 7%, > 7%, respectively, thereby providing explicit anchors for imbalance criterion scoring. Conclusions: The novel ROB tool represents a valuable instrument for quantifying the ROB of NRS that deserves further formal investigation in a larger, more diverse sample of studies. The described survey method appears an effective and efficient method to define the necessary imbalance criterion thresholds. 1. Reeves BC, Shea B, Wells GA. Classifying non-randomised studies (NRS) and the assessing the risk of bias for a systematic review. Workshop, 18th Cochrane Colloquium, Keystone 2010.

AB - Background: To date, few tools exist for assessing the risk of bias (ROB) from confounding variables in non-randomized studies (NRS). Objectives: We tested a pilot Cochrane tool for ROB to assess its applicability and value in quantifying ROB in NRS. Methods: We applied the Cochrane tool to a systematic review of NRS evaluating the effectiveness of lymph node dissection in patients undergoing radical cystectomy. Three independent review authors abstracted data on methodological quality using the Newcastle-Ottawa instrument and the Cochrane tool for assessing ROB.[1] The Cochrane tool assesses the precision, imbalance, and adjustment for each confounding variable, and overall ROB from confounding, blinding, incomplete outcome data and selective reporting, on 1 (low ROB) – 5 (high ROB) point scales. We pilot-tested the data abstraction form and developed detailed guidelines to optimize interobserver agreement. Results: Five NRS met the inclusion criteria. Confounders were identified in advance by literature search and responses to a web-based survey of eleven bladder cancer experts who were also asked to categorize what difference in the proportion of a confounding variable represented “no”, a “small”, “moderate” or “large” imbalance. To evaluate imbalance, each confounder was evaluated dichotomously (e.g. organ-confined versus non-organ-confined). Threshold values were derived using the median responses; e.g. for non-organ-confined disease, these were set at = 2%, 2.1-5%, 5.1 – 7%, > 7%, respectively, thereby providing explicit anchors for imbalance criterion scoring. Conclusions: The novel ROB tool represents a valuable instrument for quantifying the ROB of NRS that deserves further formal investigation in a larger, more diverse sample of studies. The described survey method appears an effective and efficient method to define the necessary imbalance criterion thresholds. 1. Reeves BC, Shea B, Wells GA. Classifying non-randomised studies (NRS) and the assessing the risk of bias for a systematic review. Workshop, 18th Cochrane Colloquium, Keystone 2010.

M3 - Paper

ER -