Initial experience with a pilot Cochrane tool for assessing risk of bias for non-randomized studies applying a web-based survey of content experts to derive criteria for imbalance

D. Palmer, Steven MacLennan, Mari Imamura, B. C. Reeves, A Clubb, A Duboy, Molly Nueberger, P Dahm

Research output: Contribution to conferenceUnpublished paper

Abstract

Background: To date, few tools exist for assessing the risk of bias (ROB) from confounding variables in non-randomized studies (NRS).

Objectives: We tested a pilot Cochrane tool for ROB to assess its applicability and value in quantifying ROB in NRS.

Methods: We applied the Cochrane tool to a systematic review of NRS evaluating the effectiveness of lymph node dissection in patients undergoing radical cystectomy. Three independent review authors abstracted data on methodological quality using the Newcastle-Ottawa instrument and the Cochrane tool for assessing ROB.[1] The Cochrane tool assesses the precision, imbalance, and adjustment for each confounding variable, and overall ROB from confounding, blinding, incomplete outcome data and selective reporting, on 1 (low ROB) – 5 (high ROB) point scales. We pilot-tested the data abstraction form and developed detailed guidelines to optimize interobserver agreement.

Results: Five NRS met the inclusion criteria. Confounders were identified in advance by literature search and responses to a web-based survey of eleven bladder cancer experts who were also asked to categorize what difference in the proportion of a confounding variable represented “no”, a “small”, “moderate” or “large” imbalance. To evaluate imbalance, each confounder was evaluated dichotomously (e.g. organ-confined versus non-organ-confined). Threshold values were derived using the median responses; e.g. for non-organ-confined disease, these were set at = 2%, 2.1-5%, 5.1 – 7%, > 7%, respectively, thereby providing explicit anchors for imbalance criterion scoring.

Conclusions: The novel ROB tool represents a valuable instrument for quantifying the ROB of NRS that deserves further formal investigation in a larger, more diverse sample of studies. The described survey method appears an effective and efficient method to define the necessary imbalance criterion thresholds.

1. Reeves BC, Shea B, Wells GA. Classifying non-randomised studies (NRS) and the assessing the risk of bias for a systematic review. Workshop, 18th Cochrane Colloquium, Keystone 2010.
Original languageEnglish
Publication statusPublished - Oct 2011
EventCochrane Colloquium 2011 - Madrid, Spain
Duration: 18 Oct 201122 Oct 2011

Conference

ConferenceCochrane Colloquium 2011
Country/TerritorySpain
CityMadrid
Period18/10/1122/10/11

Fingerprint

Dive into the research topics of 'Initial experience with a pilot Cochrane tool for assessing risk of bias for non-randomized studies applying a web-based survey of content experts to derive criteria for imbalance'. Together they form a unique fingerprint.

Cite this