Empirical evidence of design-related bias in studies of diagnostic tests

Jeroen G. Lijmer*, Ben Willem Mol, Siem Heisterkamp, Gouke J. Bonsel, Martin H. Prins, Jan H.P. Van Der Meulen, Patrick M.M. Bossuyt

*Corresponding author for this work

Research output: Contribution to journalArticle

1345 Citations (Scopus)

Abstract

Context: The literature contains a large number of potential biases in the evaluation of diagnostic tests. Strict application of appropriate methodological criteria would invalidate the clinical application of most study results. Objective: To empirically determine the quantitative effect of study design shortcomings on estimates of diagnostic accuracy. Design and Setting: Observational study of the methodological features of 184 original studies evaluating 218 diagnostic tests. Meta-analyses on diagnostic tests were identified through a systematic search of the literature using MEDLINE, EMBASE, and DARE databases and the Cochrane Library (1996-1997). Associations between study characteristics and estimates of diagnostic accuracy were evaluated with a regression model. Main Outcome Measures: Relative diagnostic odds ratio (RDOR), which compared the diagnostic odds ratios of studies of a given test that lacked a particular methodological feature with those without the corresponding shortcomings in design. Results: Fifteen (6.8%) of 218 evaluations met all 8 criteria; 64 (30%) met 6 or more. Studies evaluating tests in a diseased population and a separate control group overestimated the diagnostic performance compared with studies that used a clinical population (RDOR, 3.0; 95% confidence interval [CI], 2.0-4.5). Studies in which different reference tests were used for positive and negative results of the test under study overestimated the diagnostic performance compared with studies using a single reference test for all patients (RDOR, 2.2; 95% CI, 1.5-3.3). Diagnostic performance was also overestimated when the reference test was interpreted with knowledge of the test result (RDOR, 1.3; 95% CI, 1.0-1.9), when no criteria for the test were described (RDOR, 1.7; 95% CI, 1.1-2.5), and when no description of the population under study was provided (RDOR, 1.4; 95% CI, 1.1-1.7). Conclusion: These data provide empirical evidence that diagnostic studies with methodological shortcomings may overestimate the accuracy of a diagnostic test, particularly those including nonrepresentative patients or applying different reference standards.

Original languageEnglish
Pages (from-to)1061-1066
Number of pages6
JournalJournal of the American Medical Association
Volume282
Issue number11
DOIs
Publication statusPublished - 15 Sep 1999

Fingerprint Dive into the research topics of 'Empirical evidence of design-related bias in studies of diagnostic tests'. Together they form a unique fingerprint.

  • Cite this

    Lijmer, J. G., Mol, B. W., Heisterkamp, S., Bonsel, G. J., Prins, M. H., Van Der Meulen, J. H. P., & Bossuyt, P. M. M. (1999). Empirical evidence of design-related bias in studies of diagnostic tests. Journal of the American Medical Association, 282(11), 1061-1066. https://doi.org/10.1001/jama.282.11.1061