Challenging script concordance test reference standard by evidence

Do judgments by emergency medicine consultants agree with likelihood ratios?

Seyed Foad Ahmadi, Shahin Khoshkish, Kamran Soltani-Arabshahi, Peyman Hafezi-Moghadam, Golara Zahmatkesh, Parisa Heidari, Davood Baba-Beigloo, Hamid R. Baradaran, Shahram Lotfipour* (Corresponding Author)

*Corresponding author for this work

Research output: Contribution to journalArticle

6 Citations (Scopus)
4 Downloads (Pure)

Abstract

Background: We aimed to compare the clinical judgments of a reference panel of emergency medicine academic physicians against evidence-based likelihood ratios (LRs) regarding the diagnostic value of selected clinical and paraclinical findings in the context of a script concordance test (SCT).

Findings: A SCT with six scenarios and five questions per scenario was developed. Subsequently, 15 emergency medicine attending physicians (reference panel) took the test and their judgments regarding the diagnostic value of those findings for given diseases were recorded. The LRs of the same findings for the same diseases were extracted from a series of published systematic reviews. Then, the reference panel judgments were compared to evidence-based LRs. To investigate the test-retest reliability, five participants took the test one month later, and the correlation of their first and second judgments were quantified using Spearman rank-order coefficient.

In 22 out of 30 (73.3%) findings, the expert judgments were significantly different from the LRs. The differences included overestimation (30%), underestimation (30%), and judging the diagnostic value in an opposite direction (13.3%). Moreover, the score of a hypothetical test-taker was calculated to be 21.73 out of 30 if his/her answers were based on evidence-based LRs.

The test showed an acceptable test-retest reliability coefficient (Spearman coefficient: 0.83).

Conclusions: Although SCT is an interesting test to evaluate clinical decision-making in emergency medicine, our results raise concerns regarding whether the judgments of an expert panel are sufficiently valid as the reference standard for this test.

Original languageEnglish
Article number34
Pages (from-to)1-7
Number of pages7
JournalInternational Journal of Emergency Medicine
Volume7
Issue number1
DOIs
Publication statusPublished - 26 Sep 2014

Fingerprint

Emergency Medicine
Consultants
Reproducibility of Results
Physicians

Keywords

  • Clinical judgment
  • Decision-making
  • Diagnosis
  • Evidence-based medicine
  • Likelihood ratio
  • Script concordance test
  • Visual analog scales

ASJC Scopus subject areas

  • Emergency Medicine

Cite this

Challenging script concordance test reference standard by evidence : Do judgments by emergency medicine consultants agree with likelihood ratios? / Ahmadi, Seyed Foad; Khoshkish, Shahin; Soltani-Arabshahi, Kamran; Hafezi-Moghadam, Peyman; Zahmatkesh, Golara; Heidari, Parisa; Baba-Beigloo, Davood; Baradaran, Hamid R.; Lotfipour, Shahram (Corresponding Author).

In: International Journal of Emergency Medicine, Vol. 7, No. 1, 34, 26.09.2014, p. 1-7.

Research output: Contribution to journalArticle

Ahmadi, SF, Khoshkish, S, Soltani-Arabshahi, K, Hafezi-Moghadam, P, Zahmatkesh, G, Heidari, P, Baba-Beigloo, D, Baradaran, HR & Lotfipour, S 2014, 'Challenging script concordance test reference standard by evidence: Do judgments by emergency medicine consultants agree with likelihood ratios?', International Journal of Emergency Medicine, vol. 7, no. 1, 34, pp. 1-7. https://doi.org/10.1186/s12245-014-0034-3
Ahmadi, Seyed Foad ; Khoshkish, Shahin ; Soltani-Arabshahi, Kamran ; Hafezi-Moghadam, Peyman ; Zahmatkesh, Golara ; Heidari, Parisa ; Baba-Beigloo, Davood ; Baradaran, Hamid R. ; Lotfipour, Shahram. / Challenging script concordance test reference standard by evidence : Do judgments by emergency medicine consultants agree with likelihood ratios?. In: International Journal of Emergency Medicine. 2014 ; Vol. 7, No. 1. pp. 1-7.
@article{272493432ce14ce6bfa5bd185cce5862,
title = "Challenging script concordance test reference standard by evidence: Do judgments by emergency medicine consultants agree with likelihood ratios?",
abstract = "Background: We aimed to compare the clinical judgments of a reference panel of emergency medicine academic physicians against evidence-based likelihood ratios (LRs) regarding the diagnostic value of selected clinical and paraclinical findings in the context of a script concordance test (SCT).Findings: A SCT with six scenarios and five questions per scenario was developed. Subsequently, 15 emergency medicine attending physicians (reference panel) took the test and their judgments regarding the diagnostic value of those findings for given diseases were recorded. The LRs of the same findings for the same diseases were extracted from a series of published systematic reviews. Then, the reference panel judgments were compared to evidence-based LRs. To investigate the test-retest reliability, five participants took the test one month later, and the correlation of their first and second judgments were quantified using Spearman rank-order coefficient.In 22 out of 30 (73.3{\%}) findings, the expert judgments were significantly different from the LRs. The differences included overestimation (30{\%}), underestimation (30{\%}), and judging the diagnostic value in an opposite direction (13.3{\%}). Moreover, the score of a hypothetical test-taker was calculated to be 21.73 out of 30 if his/her answers were based on evidence-based LRs.The test showed an acceptable test-retest reliability coefficient (Spearman coefficient: 0.83).Conclusions: Although SCT is an interesting test to evaluate clinical decision-making in emergency medicine, our results raise concerns regarding whether the judgments of an expert panel are sufficiently valid as the reference standard for this test.",
keywords = "Clinical judgment, Decision-making, Diagnosis, Evidence-based medicine, Likelihood ratio, Script concordance test, Visual analog scales",
author = "Ahmadi, {Seyed Foad} and Shahin Khoshkish and Kamran Soltani-Arabshahi and Peyman Hafezi-Moghadam and Golara Zahmatkesh and Parisa Heidari and Davood Baba-Beigloo and Baradaran, {Hamid R.} and Shahram Lotfipour",
note = "Acknowledgements We would like to acknowledge Dr. Amir Nejati for his contributions in collecting data for this study. Sources of funding This study was the M.D. thesis of SK and was funded by Iran University of Medical Sciences. The authors have not received fund from any other source.",
year = "2014",
month = "9",
day = "26",
doi = "10.1186/s12245-014-0034-3",
language = "English",
volume = "7",
pages = "1--7",
journal = "International Journal of Emergency Medicine",
issn = "1865-1372",
publisher = "Springer London",
number = "1",

}

TY - JOUR

T1 - Challenging script concordance test reference standard by evidence

T2 - Do judgments by emergency medicine consultants agree with likelihood ratios?

AU - Ahmadi, Seyed Foad

AU - Khoshkish, Shahin

AU - Soltani-Arabshahi, Kamran

AU - Hafezi-Moghadam, Peyman

AU - Zahmatkesh, Golara

AU - Heidari, Parisa

AU - Baba-Beigloo, Davood

AU - Baradaran, Hamid R.

AU - Lotfipour, Shahram

N1 - Acknowledgements We would like to acknowledge Dr. Amir Nejati for his contributions in collecting data for this study. Sources of funding This study was the M.D. thesis of SK and was funded by Iran University of Medical Sciences. The authors have not received fund from any other source.

PY - 2014/9/26

Y1 - 2014/9/26

N2 - Background: We aimed to compare the clinical judgments of a reference panel of emergency medicine academic physicians against evidence-based likelihood ratios (LRs) regarding the diagnostic value of selected clinical and paraclinical findings in the context of a script concordance test (SCT).Findings: A SCT with six scenarios and five questions per scenario was developed. Subsequently, 15 emergency medicine attending physicians (reference panel) took the test and their judgments regarding the diagnostic value of those findings for given diseases were recorded. The LRs of the same findings for the same diseases were extracted from a series of published systematic reviews. Then, the reference panel judgments were compared to evidence-based LRs. To investigate the test-retest reliability, five participants took the test one month later, and the correlation of their first and second judgments were quantified using Spearman rank-order coefficient.In 22 out of 30 (73.3%) findings, the expert judgments were significantly different from the LRs. The differences included overestimation (30%), underestimation (30%), and judging the diagnostic value in an opposite direction (13.3%). Moreover, the score of a hypothetical test-taker was calculated to be 21.73 out of 30 if his/her answers were based on evidence-based LRs.The test showed an acceptable test-retest reliability coefficient (Spearman coefficient: 0.83).Conclusions: Although SCT is an interesting test to evaluate clinical decision-making in emergency medicine, our results raise concerns regarding whether the judgments of an expert panel are sufficiently valid as the reference standard for this test.

AB - Background: We aimed to compare the clinical judgments of a reference panel of emergency medicine academic physicians against evidence-based likelihood ratios (LRs) regarding the diagnostic value of selected clinical and paraclinical findings in the context of a script concordance test (SCT).Findings: A SCT with six scenarios and five questions per scenario was developed. Subsequently, 15 emergency medicine attending physicians (reference panel) took the test and their judgments regarding the diagnostic value of those findings for given diseases were recorded. The LRs of the same findings for the same diseases were extracted from a series of published systematic reviews. Then, the reference panel judgments were compared to evidence-based LRs. To investigate the test-retest reliability, five participants took the test one month later, and the correlation of their first and second judgments were quantified using Spearman rank-order coefficient.In 22 out of 30 (73.3%) findings, the expert judgments were significantly different from the LRs. The differences included overestimation (30%), underestimation (30%), and judging the diagnostic value in an opposite direction (13.3%). Moreover, the score of a hypothetical test-taker was calculated to be 21.73 out of 30 if his/her answers were based on evidence-based LRs.The test showed an acceptable test-retest reliability coefficient (Spearman coefficient: 0.83).Conclusions: Although SCT is an interesting test to evaluate clinical decision-making in emergency medicine, our results raise concerns regarding whether the judgments of an expert panel are sufficiently valid as the reference standard for this test.

KW - Clinical judgment

KW - Decision-making

KW - Diagnosis

KW - Evidence-based medicine

KW - Likelihood ratio

KW - Script concordance test

KW - Visual analog scales

UR - http://www.scopus.com/inward/record.url?scp=84919947701&partnerID=8YFLogxK

U2 - 10.1186/s12245-014-0034-3

DO - 10.1186/s12245-014-0034-3

M3 - Article

VL - 7

SP - 1

EP - 7

JO - International Journal of Emergency Medicine

JF - International Journal of Emergency Medicine

SN - 1865-1372

IS - 1

M1 - 34

ER -