Reporting Bias Inflates the Reputation of Medical Treatments: A Comparison of Outcomes in Clinical Trials and Online Product Reviews

Mícheál De Barra

Research output: Contribution to journalArticle

8 Citations (Scopus)
4 Downloads (Pure)

Abstract

Objectives

People often hold unduly positive expectations about the outcomes of medicines and other healthcare products. Here the following explanation is tested: people who have a positive outcome tend to tell more people about their disease/treatment than people with poor or average outcomes. Akin to the file drawer problem in science, this systematically and positively distorts the information available to others.

Method

If people with good treatment outcomes are more inclined to tell others, then they should also be more inclined to write online medical product reviews. Therefore, average treatment outcomes in these reviews should be more positive than those found in randomised controlled trials (RCTs). Data on duration of treatment and outcome (i.e., weight/cholesterol change) were extracted from user-generated health product reviews on Amazon.com and compared to RCT data for the same treatments using ANOVA. The sample included 1675 reviews of cholesterol reduction (Benecol, CholestOff) and weight loss (Orlistat) treatments and the primary outcome was cholesterol change (Bencol and CholestOff) or weight change (Orlistat).

Results

In three independent tests, average outcomes reported in the reviews were substantially more positive than the outcomes reported in the medical literature (η2 = 0.01 to 0.06; p = 0.04 to 0.001). For example, average cholesterol change following use of Benecol is −14 mg/dl in RCTs and −45 mg/dl in online reviews.

Conclusions

People with good treatment outcomes are more inclined to share information about their treatment, which distorts the information available to others. People who rely on word of mouth reputation, electronic or real life, are likely to develop unduly positive expectations.
Original languageEnglish
Pages (from-to)248-255
Number of pages8
JournalSocial Science & Medicine
Volume177
Early online date10 Feb 2017
DOIs
Publication statusPublished - Mar 2017

Fingerprint

physician's care
reputation
Clinical Trials
Cholesterol
trend
Randomized Controlled Trials
available information
Weights and Measures
Weight Loss
Analysis of Variance
Medical Treatment
medicine
electronics
Disease
Delivery of Health Care
Health
science
health
Treatment Outcome
Randomized Controlled Trial

Keywords

  • health informatics
  • eHealth
  • medical overuse
  • word of mouth
  • cultural evolution

Cite this

Reporting Bias Inflates the Reputation of Medical Treatments : A Comparison of Outcomes in Clinical Trials and Online Product Reviews . / De Barra, Mícheál.

In: Social Science & Medicine, Vol. 177, 03.2017, p. 248-255.

Research output: Contribution to journalArticle

@article{11f7a40a62cf476db289773b5c792856,
title = "Reporting Bias Inflates the Reputation of Medical Treatments: A Comparison of Outcomes in Clinical Trials and Online Product Reviews",
abstract = "ObjectivesPeople often hold unduly positive expectations about the outcomes of medicines and other healthcare products. Here the following explanation is tested: people who have a positive outcome tend to tell more people about their disease/treatment than people with poor or average outcomes. Akin to the file drawer problem in science, this systematically and positively distorts the information available to others.MethodIf people with good treatment outcomes are more inclined to tell others, then they should also be more inclined to write online medical product reviews. Therefore, average treatment outcomes in these reviews should be more positive than those found in randomised controlled trials (RCTs). Data on duration of treatment and outcome (i.e., weight/cholesterol change) were extracted from user-generated health product reviews on Amazon.com and compared to RCT data for the same treatments using ANOVA. The sample included 1675 reviews of cholesterol reduction (Benecol, CholestOff) and weight loss (Orlistat) treatments and the primary outcome was cholesterol change (Bencol and CholestOff) or weight change (Orlistat).ResultsIn three independent tests, average outcomes reported in the reviews were substantially more positive than the outcomes reported in the medical literature (η2 = 0.01 to 0.06; p = 0.04 to 0.001). For example, average cholesterol change following use of Benecol is −14 mg/dl in RCTs and −45 mg/dl in online reviews.ConclusionsPeople with good treatment outcomes are more inclined to share information about their treatment, which distorts the information available to others. People who rely on word of mouth reputation, electronic or real life, are likely to develop unduly positive expectations.",
keywords = "health informatics, eHealth, medical overuse, word of mouth, cultural evolution",
author = "{De Barra}, M{\'i}che{\'a}l",
note = "This research was supported in part by the Prof Roy Weir Career Development Fellowship. Kimmo Eriksson and Pontus Strimling encouraged me develop the ideas in this manuscript. Turu Stadler advised on some of the statistical analyses and Marie Johnston made many useful suggestions.",
year = "2017",
month = "3",
doi = "10.1016/j.socscimed.2017.01.033",
language = "English",
volume = "177",
pages = "248--255",
journal = "Social Science & Medicine",
issn = "0277-9536",
publisher = "Elsevier Limited",

}

TY - JOUR

T1 - Reporting Bias Inflates the Reputation of Medical Treatments

T2 - A Comparison of Outcomes in Clinical Trials and Online Product Reviews

AU - De Barra, Mícheál

N1 - This research was supported in part by the Prof Roy Weir Career Development Fellowship. Kimmo Eriksson and Pontus Strimling encouraged me develop the ideas in this manuscript. Turu Stadler advised on some of the statistical analyses and Marie Johnston made many useful suggestions.

PY - 2017/3

Y1 - 2017/3

N2 - ObjectivesPeople often hold unduly positive expectations about the outcomes of medicines and other healthcare products. Here the following explanation is tested: people who have a positive outcome tend to tell more people about their disease/treatment than people with poor or average outcomes. Akin to the file drawer problem in science, this systematically and positively distorts the information available to others.MethodIf people with good treatment outcomes are more inclined to tell others, then they should also be more inclined to write online medical product reviews. Therefore, average treatment outcomes in these reviews should be more positive than those found in randomised controlled trials (RCTs). Data on duration of treatment and outcome (i.e., weight/cholesterol change) were extracted from user-generated health product reviews on Amazon.com and compared to RCT data for the same treatments using ANOVA. The sample included 1675 reviews of cholesterol reduction (Benecol, CholestOff) and weight loss (Orlistat) treatments and the primary outcome was cholesterol change (Bencol and CholestOff) or weight change (Orlistat).ResultsIn three independent tests, average outcomes reported in the reviews were substantially more positive than the outcomes reported in the medical literature (η2 = 0.01 to 0.06; p = 0.04 to 0.001). For example, average cholesterol change following use of Benecol is −14 mg/dl in RCTs and −45 mg/dl in online reviews.ConclusionsPeople with good treatment outcomes are more inclined to share information about their treatment, which distorts the information available to others. People who rely on word of mouth reputation, electronic or real life, are likely to develop unduly positive expectations.

AB - ObjectivesPeople often hold unduly positive expectations about the outcomes of medicines and other healthcare products. Here the following explanation is tested: people who have a positive outcome tend to tell more people about their disease/treatment than people with poor or average outcomes. Akin to the file drawer problem in science, this systematically and positively distorts the information available to others.MethodIf people with good treatment outcomes are more inclined to tell others, then they should also be more inclined to write online medical product reviews. Therefore, average treatment outcomes in these reviews should be more positive than those found in randomised controlled trials (RCTs). Data on duration of treatment and outcome (i.e., weight/cholesterol change) were extracted from user-generated health product reviews on Amazon.com and compared to RCT data for the same treatments using ANOVA. The sample included 1675 reviews of cholesterol reduction (Benecol, CholestOff) and weight loss (Orlistat) treatments and the primary outcome was cholesterol change (Bencol and CholestOff) or weight change (Orlistat).ResultsIn three independent tests, average outcomes reported in the reviews were substantially more positive than the outcomes reported in the medical literature (η2 = 0.01 to 0.06; p = 0.04 to 0.001). For example, average cholesterol change following use of Benecol is −14 mg/dl in RCTs and −45 mg/dl in online reviews.ConclusionsPeople with good treatment outcomes are more inclined to share information about their treatment, which distorts the information available to others. People who rely on word of mouth reputation, electronic or real life, are likely to develop unduly positive expectations.

KW - health informatics

KW - eHealth

KW - medical overuse

KW - word of mouth

KW - cultural evolution

U2 - 10.1016/j.socscimed.2017.01.033

DO - 10.1016/j.socscimed.2017.01.033

M3 - Article

VL - 177

SP - 248

EP - 255

JO - Social Science & Medicine

JF - Social Science & Medicine

SN - 0277-9536

ER -