Sequential testing in a high stakes OSCE: determining number of screening tests

Graeme P. Currie, Selvaraj Sivasubramaniam, Jennifer Cleland

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

OBJECTIVE: The sequential objective structured clinical exam (OSCE) is a stand-alone variation of the traditional OSCE whereby all students sit a screening test. Those who pass this initial assessment undergo no further testing while weakly performing students sit an additional (sequential) test to determine their overall pass/fail status. Our aim was to determine outcomes of adopting a sequential OSCE approach using different numbers of screening stations and pass marks.

METHOD: We carried out a retrospective, observational study of anonymised databases of two cohorts of student outcomes from the final OSCE examination at the University of Aberdeen Medical School. Data were accessed for students (n = 388) who sat the exam in the years 2013-2014. We used Stata simulate program to compare outcomes - in terms of sensitivity and specificity - across 5000 random selections of 6-14 OSCE stations using random selections of groups of 100 students (with different screening test pass marks) versus those obtained across 15 stations.

RESULTS: Across 6-14 stations, the sensitivity was ≥87% in 2013 and ≥84% in 2014 while the specificity ranged from 60% to 100% in both years. Specificity generally increased as the number of screening stations increased (with concomitant narrowing of the 95% confidence interval), while sensitivity varied between 84 and 98%. Similar sensitivities and specificities were found with screening pass marks of +1, +2 and +3 standard errors of measurement (SEM). Eight stations as a screening test appeared to be a reasonable compromise in terms of high sensitivity (88-89%) and specificity (83-86%).

CONCLUSION: This research extends current sequential OSCE literature using a novel and robust approach to identify the "ideal" in terms of number of screening stations and pass mark. We discuss the educational and resource implications of our findings and make recommendations for the use of the sequential OSCE in medical education.

Original languageEnglish
Pages (from-to)708-714
Number of pages7
JournalMedical Teacher
Volume38
Issue number7
Early online date16 Oct 2015
DOIs
Publication statusPublished - 2016

Fingerprint

student
compromise
confidence
examination
resources
school
education
Group
literature

Cite this

Sequential testing in a high stakes OSCE : determining number of screening tests. / Currie, Graeme P.; Sivasubramaniam, Selvaraj; Cleland, Jennifer.

In: Medical Teacher, Vol. 38, No. 7, 2016, p. 708-714.

Research output: Contribution to journalArticle

@article{756390184ba740ea89c99ee38614baae,
title = "Sequential testing in a high stakes OSCE: determining number of screening tests",
abstract = "OBJECTIVE: The sequential objective structured clinical exam (OSCE) is a stand-alone variation of the traditional OSCE whereby all students sit a screening test. Those who pass this initial assessment undergo no further testing while weakly performing students sit an additional (sequential) test to determine their overall pass/fail status. Our aim was to determine outcomes of adopting a sequential OSCE approach using different numbers of screening stations and pass marks.METHOD: We carried out a retrospective, observational study of anonymised databases of two cohorts of student outcomes from the final OSCE examination at the University of Aberdeen Medical School. Data were accessed for students (n = 388) who sat the exam in the years 2013-2014. We used Stata simulate program to compare outcomes - in terms of sensitivity and specificity - across 5000 random selections of 6-14 OSCE stations using random selections of groups of 100 students (with different screening test pass marks) versus those obtained across 15 stations.RESULTS: Across 6-14 stations, the sensitivity was ≥87{\%} in 2013 and ≥84{\%} in 2014 while the specificity ranged from 60{\%} to 100{\%} in both years. Specificity generally increased as the number of screening stations increased (with concomitant narrowing of the 95{\%} confidence interval), while sensitivity varied between 84 and 98{\%}. Similar sensitivities and specificities were found with screening pass marks of +1, +2 and +3 standard errors of measurement (SEM). Eight stations as a screening test appeared to be a reasonable compromise in terms of high sensitivity (88-89{\%}) and specificity (83-86{\%}).CONCLUSION: This research extends current sequential OSCE literature using a novel and robust approach to identify the {"}ideal{"} in terms of number of screening stations and pass mark. We discuss the educational and resource implications of our findings and make recommendations for the use of the sequential OSCE in medical education.",
author = "Currie, {Graeme P.} and Selvaraj Sivasubramaniam and Jennifer Cleland",
year = "2016",
doi = "10.3109/0142159X.2015.1079309",
language = "English",
volume = "38",
pages = "708--714",
journal = "Medical Teacher",
issn = "0142-159X",
publisher = "Informa Healthcare",
number = "7",

}

TY - JOUR

T1 - Sequential testing in a high stakes OSCE

T2 - determining number of screening tests

AU - Currie, Graeme P.

AU - Sivasubramaniam, Selvaraj

AU - Cleland, Jennifer

PY - 2016

Y1 - 2016

N2 - OBJECTIVE: The sequential objective structured clinical exam (OSCE) is a stand-alone variation of the traditional OSCE whereby all students sit a screening test. Those who pass this initial assessment undergo no further testing while weakly performing students sit an additional (sequential) test to determine their overall pass/fail status. Our aim was to determine outcomes of adopting a sequential OSCE approach using different numbers of screening stations and pass marks.METHOD: We carried out a retrospective, observational study of anonymised databases of two cohorts of student outcomes from the final OSCE examination at the University of Aberdeen Medical School. Data were accessed for students (n = 388) who sat the exam in the years 2013-2014. We used Stata simulate program to compare outcomes - in terms of sensitivity and specificity - across 5000 random selections of 6-14 OSCE stations using random selections of groups of 100 students (with different screening test pass marks) versus those obtained across 15 stations.RESULTS: Across 6-14 stations, the sensitivity was ≥87% in 2013 and ≥84% in 2014 while the specificity ranged from 60% to 100% in both years. Specificity generally increased as the number of screening stations increased (with concomitant narrowing of the 95% confidence interval), while sensitivity varied between 84 and 98%. Similar sensitivities and specificities were found with screening pass marks of +1, +2 and +3 standard errors of measurement (SEM). Eight stations as a screening test appeared to be a reasonable compromise in terms of high sensitivity (88-89%) and specificity (83-86%).CONCLUSION: This research extends current sequential OSCE literature using a novel and robust approach to identify the "ideal" in terms of number of screening stations and pass mark. We discuss the educational and resource implications of our findings and make recommendations for the use of the sequential OSCE in medical education.

AB - OBJECTIVE: The sequential objective structured clinical exam (OSCE) is a stand-alone variation of the traditional OSCE whereby all students sit a screening test. Those who pass this initial assessment undergo no further testing while weakly performing students sit an additional (sequential) test to determine their overall pass/fail status. Our aim was to determine outcomes of adopting a sequential OSCE approach using different numbers of screening stations and pass marks.METHOD: We carried out a retrospective, observational study of anonymised databases of two cohorts of student outcomes from the final OSCE examination at the University of Aberdeen Medical School. Data were accessed for students (n = 388) who sat the exam in the years 2013-2014. We used Stata simulate program to compare outcomes - in terms of sensitivity and specificity - across 5000 random selections of 6-14 OSCE stations using random selections of groups of 100 students (with different screening test pass marks) versus those obtained across 15 stations.RESULTS: Across 6-14 stations, the sensitivity was ≥87% in 2013 and ≥84% in 2014 while the specificity ranged from 60% to 100% in both years. Specificity generally increased as the number of screening stations increased (with concomitant narrowing of the 95% confidence interval), while sensitivity varied between 84 and 98%. Similar sensitivities and specificities were found with screening pass marks of +1, +2 and +3 standard errors of measurement (SEM). Eight stations as a screening test appeared to be a reasonable compromise in terms of high sensitivity (88-89%) and specificity (83-86%).CONCLUSION: This research extends current sequential OSCE literature using a novel and robust approach to identify the "ideal" in terms of number of screening stations and pass mark. We discuss the educational and resource implications of our findings and make recommendations for the use of the sequential OSCE in medical education.

U2 - 10.3109/0142159X.2015.1079309

DO - 10.3109/0142159X.2015.1079309

M3 - Article

VL - 38

SP - 708

EP - 714

JO - Medical Teacher

JF - Medical Teacher

SN - 0142-159X

IS - 7

ER -