Opening the black box of selection

Sanne Schreurs (Corresponding Author), Kitty Cleutjens, Carlos F Collares, Jennifer Cleland, Mirjam G A Oude Egbrink

Research output: Contribution to journalArticle

1 Downloads (Pure)

Abstract

Medical school selection is currently in the paradoxical situation in which selection tools may predict study outcomes, but which constructs are actually doing the predicting is unknown (the 'black box of selection'). Therefore, our research focused on those constructs, answering the question: do the internal structures of the tests in an outcome-based selection procedure reflect the content that was intended to be measured? Downing's validity framework was applied to organize evidence for construct validity, focusing on evidence related to content and internal structure. The applied selection procedure was a multi-tool, CanMEDS-based procedure comprised of a video-based situational judgement test (focused on (inter)personal competencies), and a written aptitude test (reflecting a broader array of CanMEDS competencies). First, we examined content-related evidence pertaining to the creation and application of the competency-based selection blueprint and found that the set-up of the selection procedure was a robust, transparent and replicable process. Second, the internal structure of the selection tests was investigated by connecting applicants' performance on the selection tests to the predetermined blueprint using cognitive diagnostic modeling. The data indicate 89% overlap between the expected and measured constructs. Our results support the notion that the focus placed on creating the right content and following a competency-blueprint was effective in terms of internal structure: most items measured what they were intended to measure. This way of linking a predetermined blueprint to the applicants' results sheds light into the 'black box of selection' and can be used to support the construct validity of selection procedures.

Original languageEnglish
Pages (from-to)1-20
Number of pages20
JournalAdvances in Health Sciences Education
Early online date9 Oct 2019
DOIs
Publication statusE-pub ahead of print - 9 Oct 2019

Fingerprint

selection procedure
construct validity
applicant
evidence
aptitude test
diagnostic
video
school
performance

Keywords

  • Selection
  • Medical School
  • Admissions
  • Construct validity

Cite this

Schreurs, S., Cleutjens, K., Collares, C. F., Cleland, J., & Oude Egbrink, M. G. A. (2019). Opening the black box of selection. Advances in Health Sciences Education, 1-20. https://doi.org/10.1007/s10459-019-09925-1

Opening the black box of selection. / Schreurs, Sanne (Corresponding Author); Cleutjens, Kitty; Collares, Carlos F; Cleland, Jennifer; Oude Egbrink, Mirjam G A.

In: Advances in Health Sciences Education, 09.10.2019, p. 1-20.

Research output: Contribution to journalArticle

Schreurs, Sanne ; Cleutjens, Kitty ; Collares, Carlos F ; Cleland, Jennifer ; Oude Egbrink, Mirjam G A. / Opening the black box of selection. In: Advances in Health Sciences Education. 2019 ; pp. 1-20.
@article{ead8ade4f1c8437fab75145ffd71d418,
title = "Opening the black box of selection",
abstract = "Medical school selection is currently in the paradoxical situation in which selection tools may predict study outcomes, but which constructs are actually doing the predicting is unknown (the 'black box of selection'). Therefore, our research focused on those constructs, answering the question: do the internal structures of the tests in an outcome-based selection procedure reflect the content that was intended to be measured? Downing's validity framework was applied to organize evidence for construct validity, focusing on evidence related to content and internal structure. The applied selection procedure was a multi-tool, CanMEDS-based procedure comprised of a video-based situational judgement test (focused on (inter)personal competencies), and a written aptitude test (reflecting a broader array of CanMEDS competencies). First, we examined content-related evidence pertaining to the creation and application of the competency-based selection blueprint and found that the set-up of the selection procedure was a robust, transparent and replicable process. Second, the internal structure of the selection tests was investigated by connecting applicants' performance on the selection tests to the predetermined blueprint using cognitive diagnostic modeling. The data indicate 89{\%} overlap between the expected and measured constructs. Our results support the notion that the focus placed on creating the right content and following a competency-blueprint was effective in terms of internal structure: most items measured what they were intended to measure. This way of linking a predetermined blueprint to the applicants' results sheds light into the 'black box of selection' and can be used to support the construct validity of selection procedures.",
keywords = "Selection, Medical School, Admissions, Construct validity",
author = "Sanne Schreurs and Kitty Cleutjens and Collares, {Carlos F} and Jennifer Cleland and {Oude Egbrink}, {Mirjam G A}",
note = "Authors contribution SS, KC, MoE and JC were all involved in the conception of the study. KC retrieved the data, SS anonymized and organized the data. SS, KC and CC conducted the data analyses. All authors were involved in writing and revising the article.",
year = "2019",
month = "10",
day = "9",
doi = "10.1007/s10459-019-09925-1",
language = "English",
pages = "1--20",
journal = "Advances in Health Sciences Education",
issn = "1382-4996",
publisher = "Springer Netherlands",

}

TY - JOUR

T1 - Opening the black box of selection

AU - Schreurs, Sanne

AU - Cleutjens, Kitty

AU - Collares, Carlos F

AU - Cleland, Jennifer

AU - Oude Egbrink, Mirjam G A

N1 - Authors contribution SS, KC, MoE and JC were all involved in the conception of the study. KC retrieved the data, SS anonymized and organized the data. SS, KC and CC conducted the data analyses. All authors were involved in writing and revising the article.

PY - 2019/10/9

Y1 - 2019/10/9

N2 - Medical school selection is currently in the paradoxical situation in which selection tools may predict study outcomes, but which constructs are actually doing the predicting is unknown (the 'black box of selection'). Therefore, our research focused on those constructs, answering the question: do the internal structures of the tests in an outcome-based selection procedure reflect the content that was intended to be measured? Downing's validity framework was applied to organize evidence for construct validity, focusing on evidence related to content and internal structure. The applied selection procedure was a multi-tool, CanMEDS-based procedure comprised of a video-based situational judgement test (focused on (inter)personal competencies), and a written aptitude test (reflecting a broader array of CanMEDS competencies). First, we examined content-related evidence pertaining to the creation and application of the competency-based selection blueprint and found that the set-up of the selection procedure was a robust, transparent and replicable process. Second, the internal structure of the selection tests was investigated by connecting applicants' performance on the selection tests to the predetermined blueprint using cognitive diagnostic modeling. The data indicate 89% overlap between the expected and measured constructs. Our results support the notion that the focus placed on creating the right content and following a competency-blueprint was effective in terms of internal structure: most items measured what they were intended to measure. This way of linking a predetermined blueprint to the applicants' results sheds light into the 'black box of selection' and can be used to support the construct validity of selection procedures.

AB - Medical school selection is currently in the paradoxical situation in which selection tools may predict study outcomes, but which constructs are actually doing the predicting is unknown (the 'black box of selection'). Therefore, our research focused on those constructs, answering the question: do the internal structures of the tests in an outcome-based selection procedure reflect the content that was intended to be measured? Downing's validity framework was applied to organize evidence for construct validity, focusing on evidence related to content and internal structure. The applied selection procedure was a multi-tool, CanMEDS-based procedure comprised of a video-based situational judgement test (focused on (inter)personal competencies), and a written aptitude test (reflecting a broader array of CanMEDS competencies). First, we examined content-related evidence pertaining to the creation and application of the competency-based selection blueprint and found that the set-up of the selection procedure was a robust, transparent and replicable process. Second, the internal structure of the selection tests was investigated by connecting applicants' performance on the selection tests to the predetermined blueprint using cognitive diagnostic modeling. The data indicate 89% overlap between the expected and measured constructs. Our results support the notion that the focus placed on creating the right content and following a competency-blueprint was effective in terms of internal structure: most items measured what they were intended to measure. This way of linking a predetermined blueprint to the applicants' results sheds light into the 'black box of selection' and can be used to support the construct validity of selection procedures.

KW - Selection

KW - Medical School

KW - Admissions

KW - Construct validity

U2 - 10.1007/s10459-019-09925-1

DO - 10.1007/s10459-019-09925-1

M3 - Article

C2 - 31598883

SP - 1

EP - 20

JO - Advances in Health Sciences Education

JF - Advances in Health Sciences Education

SN - 1382-4996

ER -