Mode and Frame Matter

assessing the impact of survey mode and sample frame in choice experiments

Verity Watson* (Corresponding Author), Terry Porteous, Tim Bolt, Mandy Ryan

*Corresponding author for this work

Research output: Contribution to journalArticle

Abstract

Background. Choice experiments (CE) are applied in health economics to elicit public preferences and willingness to pay (WTP). CEs are frequently administered as Internet-based surveys. Internet surveys have recognized advantages, but concerns exist about the representativeness of Internet samples, data quality, and the impact on elicited values. Aim. We conducted the first study in health comparing an Internet-based CE survey with the more traditional general population mail survey. We also compared the Internet-based and mail CE surveys with computer-assisted personal interviews (CAPIs), which are commonly used to elicit health state valuations. Methods. Two separate samples were drawn from 2 United Kingdom (UK) volunteer Internet panels (IPs), CAPIs were undertaken with respondents sampled from UK Census Output Areas, and mail surveys were sent to UK households drawn from the postcode address file (PAF). Each mode received more than 1000 respondents. We compared modes and frames using objective measures (response rate, sample representativeness of the UK population, elicited values, theoretical validity, and cost per response) and subjective/self-reported measures (time taken to complete the study, perceived study consequentiality, and stated attribute nonattendance). This study intentionally confounded the survey modes and sample frame by choosing sample frames that are typically used by researchers for each mode. Results. Estimated WTP differs across mode-frame pairs. On most measures, CAPIs dominated. They are more expensive, however. On all measures, except response rates, Internet surveys dominated the mail survey. They were also cheaper. Conclusion. Researchers using IPs should pay attention to response rates and be aware that the quality of IPs differs. Given the importance of perceived consequentiality and attribute attendance in CEs, future research should address their impact across modes and frames.
Original languageEnglish
Pages (from-to)827-841
Number of pages15
JournalMedical Decision Making
Volume39
Issue number7
Early online date15 Sep 2019
DOIs
Publication statusPublished - Oct 2019

Fingerprint

Internet
Postal Service
Microcomputers
Interviews
Health
Surveys and Questionnaires
Research Personnel
Censuses
Population
Volunteers
Economics
Costs and Cost Analysis
United Kingdom

Keywords

  • survey mode
  • sample frame
  • choice experiment
  • health care
  • RESPONSE RATES
  • MEDICINES
  • ADHERENCE
  • ONLINE
  • MAIL
  • QUESTIONNAIRES
  • PREFERENCES
  • FORMATS
  • BELIEFS
  • ASTHMA

Cite this

@article{826e5b99311447b88c43aac5e86017a3,
title = "Mode and Frame Matter: assessing the impact of survey mode and sample frame in choice experiments",
abstract = "Background. Choice experiments (CE) are applied in health economics to elicit public preferences and willingness to pay (WTP). CEs are frequently administered as Internet-based surveys. Internet surveys have recognized advantages, but concerns exist about the representativeness of Internet samples, data quality, and the impact on elicited values. Aim. We conducted the first study in health comparing an Internet-based CE survey with the more traditional general population mail survey. We also compared the Internet-based and mail CE surveys with computer-assisted personal interviews (CAPIs), which are commonly used to elicit health state valuations. Methods. Two separate samples were drawn from 2 United Kingdom (UK) volunteer Internet panels (IPs), CAPIs were undertaken with respondents sampled from UK Census Output Areas, and mail surveys were sent to UK households drawn from the postcode address file (PAF). Each mode received more than 1000 respondents. We compared modes and frames using objective measures (response rate, sample representativeness of the UK population, elicited values, theoretical validity, and cost per response) and subjective/self-reported measures (time taken to complete the study, perceived study consequentiality, and stated attribute nonattendance). This study intentionally confounded the survey modes and sample frame by choosing sample frames that are typically used by researchers for each mode. Results. Estimated WTP differs across mode-frame pairs. On most measures, CAPIs dominated. They are more expensive, however. On all measures, except response rates, Internet surveys dominated the mail survey. They were also cheaper. Conclusion. Researchers using IPs should pay attention to response rates and be aware that the quality of IPs differs. Given the importance of perceived consequentiality and attribute attendance in CEs, future research should address their impact across modes and frames.",
keywords = "survey mode, sample frame, choice experiment, health care, RESPONSE RATES, MEDICINES, ADHERENCE, ONLINE, MAIL, QUESTIONNAIRES, PREFERENCES, FORMATS, BELIEFS, ASTHMA",
author = "Verity Watson and Terry Porteous and Tim Bolt and Mandy Ryan",
note = "This work was funded by the Medical Research Council MR/J004812/1. The Health Economics Research Unit is funded by the Chief Scientists Office of the Scottish Government Health and Social Care Directorates.",
year = "2019",
month = "10",
doi = "10.1177/0272989X19871035",
language = "English",
volume = "39",
pages = "827--841",
journal = "Medical Decision Making",
issn = "0272-989X",
publisher = "SAGE Publications Inc.",
number = "7",

}

TY - JOUR

T1 - Mode and Frame Matter

T2 - assessing the impact of survey mode and sample frame in choice experiments

AU - Watson, Verity

AU - Porteous, Terry

AU - Bolt, Tim

AU - Ryan, Mandy

N1 - This work was funded by the Medical Research Council MR/J004812/1. The Health Economics Research Unit is funded by the Chief Scientists Office of the Scottish Government Health and Social Care Directorates.

PY - 2019/10

Y1 - 2019/10

N2 - Background. Choice experiments (CE) are applied in health economics to elicit public preferences and willingness to pay (WTP). CEs are frequently administered as Internet-based surveys. Internet surveys have recognized advantages, but concerns exist about the representativeness of Internet samples, data quality, and the impact on elicited values. Aim. We conducted the first study in health comparing an Internet-based CE survey with the more traditional general population mail survey. We also compared the Internet-based and mail CE surveys with computer-assisted personal interviews (CAPIs), which are commonly used to elicit health state valuations. Methods. Two separate samples were drawn from 2 United Kingdom (UK) volunteer Internet panels (IPs), CAPIs were undertaken with respondents sampled from UK Census Output Areas, and mail surveys were sent to UK households drawn from the postcode address file (PAF). Each mode received more than 1000 respondents. We compared modes and frames using objective measures (response rate, sample representativeness of the UK population, elicited values, theoretical validity, and cost per response) and subjective/self-reported measures (time taken to complete the study, perceived study consequentiality, and stated attribute nonattendance). This study intentionally confounded the survey modes and sample frame by choosing sample frames that are typically used by researchers for each mode. Results. Estimated WTP differs across mode-frame pairs. On most measures, CAPIs dominated. They are more expensive, however. On all measures, except response rates, Internet surveys dominated the mail survey. They were also cheaper. Conclusion. Researchers using IPs should pay attention to response rates and be aware that the quality of IPs differs. Given the importance of perceived consequentiality and attribute attendance in CEs, future research should address their impact across modes and frames.

AB - Background. Choice experiments (CE) are applied in health economics to elicit public preferences and willingness to pay (WTP). CEs are frequently administered as Internet-based surveys. Internet surveys have recognized advantages, but concerns exist about the representativeness of Internet samples, data quality, and the impact on elicited values. Aim. We conducted the first study in health comparing an Internet-based CE survey with the more traditional general population mail survey. We also compared the Internet-based and mail CE surveys with computer-assisted personal interviews (CAPIs), which are commonly used to elicit health state valuations. Methods. Two separate samples were drawn from 2 United Kingdom (UK) volunteer Internet panels (IPs), CAPIs were undertaken with respondents sampled from UK Census Output Areas, and mail surveys were sent to UK households drawn from the postcode address file (PAF). Each mode received more than 1000 respondents. We compared modes and frames using objective measures (response rate, sample representativeness of the UK population, elicited values, theoretical validity, and cost per response) and subjective/self-reported measures (time taken to complete the study, perceived study consequentiality, and stated attribute nonattendance). This study intentionally confounded the survey modes and sample frame by choosing sample frames that are typically used by researchers for each mode. Results. Estimated WTP differs across mode-frame pairs. On most measures, CAPIs dominated. They are more expensive, however. On all measures, except response rates, Internet surveys dominated the mail survey. They were also cheaper. Conclusion. Researchers using IPs should pay attention to response rates and be aware that the quality of IPs differs. Given the importance of perceived consequentiality and attribute attendance in CEs, future research should address their impact across modes and frames.

KW - survey mode

KW - sample frame

KW - choice experiment

KW - health care

KW - RESPONSE RATES

KW - MEDICINES

KW - ADHERENCE

KW - ONLINE

KW - MAIL

KW - QUESTIONNAIRES

KW - PREFERENCES

KW - FORMATS

KW - BELIEFS

KW - ASTHMA

U2 - 10.1177/0272989X19871035

DO - 10.1177/0272989X19871035

M3 - Article

VL - 39

SP - 827

EP - 841

JO - Medical Decision Making

JF - Medical Decision Making

SN - 0272-989X

IS - 7

ER -