How are debriefing questions used in health discrete choice experiments? An online survey

OBJECTIVES: Debriefing questions can assess if respondents understand discrete choice experiments (DCEs) and are answering in a way consistent with theories of decision making and utility maximization. Nevertheless, there is limited literature about how often debriefing questions are included or how the results are used in health economics. The aim of this study was to conduct a survey of the frequency, type, and analysis of debriefing questions in health DCEs.

METHODS: We conducted an online survey of authors of published health DCEs, asking about their use of debriefing questions, including frequency, type, and analysis. We descriptively analyzed the sample characteristics and responses. Free-text questions were analyzed with qualitative thematic analysis.

RESULTS: We received 70 responses (43% response rate), of which 50% reported using debriefing questions. They were most commonly designed to assess difficulty (91%), understanding (49%), and attribute nonattendance (31%) rather than learning effects (3%) or monotonicity (11%). On average, 37% of debriefing questions were analyzed (range, 0% to 69%), and the results were used <50% of the time, usually to exclude respondents or interpret overall results. Researcher experience or confidence with DCEs did not affect their use of debriefing questions.

CONCLUSIONS: These results suggest that although half of researchers conducting health DCEs use debriefing questions, many do not analyze, use, or report the responses. Given the additional respondent burden, there is a need for reliable and valid debriefing questions. In the meantime, the inclusion, analysis, and reporting of debriefing questions should be carefully considered before DCE implementation.