Abstract
Objective:
The objectives of this nested study were to (1) assess whether changes in scores between rounds altered the final degree of consensus achieved in three Delphi surveys conducted as part of COS development projects (anal, gastric and prostate cancer), and (2) explore participants’ reasons for changing scores between rounds
Study design and setting:
All Delphi surveys were conducted online using DelphiManager software and included healthcare professionals and patient participants. Participants were invited to give a free-text reason whenever they changed their score across an importance threshold in a 1-9 Likert scale (1-3 not important, 4- 5 important, 7-9 critically important). Reasons for score change were coded by four researchers independently using an inductive-iterative approach.
Results:
In all three Delphi surveys, the number of outcomes reaching criteria for consensus was greater in R2 than R1. Twelve themes and 23 sub-themes emerged from 2298 discrete reasons given for score change. The most common reasons for change were ‘time to reflect’ (482 responses, 23%) and
vicarious thinking (424, 21%), with 68% (291) of vicarious thinking attributed to seeing other participants scores.
Conclusion:
Our findings support conducting a Delphi survey over the use of a single questionnaire where building consensus is the objective. Time to reflect and vicarious thinking, facilitated by seeing other participants scores, were important drivers of score change. How results are presented to participants between rounds and the duration of and time between rounds in a Delphi survey may
therefore influence the results and should be clearly reported.
The objectives of this nested study were to (1) assess whether changes in scores between rounds altered the final degree of consensus achieved in three Delphi surveys conducted as part of COS development projects (anal, gastric and prostate cancer), and (2) explore participants’ reasons for changing scores between rounds
Study design and setting:
All Delphi surveys were conducted online using DelphiManager software and included healthcare professionals and patient participants. Participants were invited to give a free-text reason whenever they changed their score across an importance threshold in a 1-9 Likert scale (1-3 not important, 4- 5 important, 7-9 critically important). Reasons for score change were coded by four researchers independently using an inductive-iterative approach.
Results:
In all three Delphi surveys, the number of outcomes reaching criteria for consensus was greater in R2 than R1. Twelve themes and 23 sub-themes emerged from 2298 discrete reasons given for score change. The most common reasons for change were ‘time to reflect’ (482 responses, 23%) and
vicarious thinking (424, 21%), with 68% (291) of vicarious thinking attributed to seeing other participants scores.
Conclusion:
Our findings support conducting a Delphi survey over the use of a single questionnaire where building consensus is the objective. Time to reflect and vicarious thinking, facilitated by seeing other participants scores, were important drivers of score change. How results are presented to participants between rounds and the duration of and time between rounds in a Delphi survey may
therefore influence the results and should be clearly reported.
| Original language | English |
|---|---|
| Pages (from-to) | 118-129 |
| Number of pages | 12 |
| Journal | Journal of Clinical Epidemiology |
| Volume | 128 |
| Early online date | 30 Sep 2020 |
| DOIs | |
| Publication status | Published - 1 Dec 2020 |
Keywords
- Core outcome set development
- Delphi survey
- consensus
- stakeholders
- feedback methods
Access to Document
- Fish_etal_Vicarious_thinking_VOR
0895-4356/ 2020 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/ licenses/by-nc-nd/4.0/).