Systematic review and statistical analysis of the integrity of 33 randomized controlled trials

Mark J Bolland, Alison Avenell, Greg D Gamble, Andrew Grey

Research output: Contribution to journalArticle

24 Citations (Scopus)

Abstract

Background: Statistical techniques can investigate data integrity in randomized controlled trials (RCTs). We systematically reviewed and analyzed all human RCTs undertaken by a group of researchers, about which concerns have been raised.

Methods: We compared observed distributions of p values for between-groups differences in baseline variables, for standardized sample means for continuous baseline variables, and for differences in treatment group participant numbers with the expected distributions. We assessed productivity, recruitment rates, outcome data, textual consistency, and ethical oversight.

Results: The researchers were remarkably productive, publishing 33 RCTs over 15 years involving large numbers of older patients with substantial comorbidity, recruited over very short periods. Treatment groups were improbably similar. The distribution of p values for differences in baseline characteristics differed markedly from the expected uniform distribution (p = 5.2 × 10−82). The distribution of standardized sample means for baseline continuous variables and the differences between participant numbers in randomized groups also differed markedly from the expected distributions (p = 4.3 × 10−4p = 1.5 × 10−5, respectively). Outcomes were remarkably positive, with very low mortality and study withdrawals despite substantial comorbidity. There were very large reductions in hip fracture incidence, regardless of intervention (relative risk 0.22, 95% confidence interval 0.15–0.31, p < 0.0001, range of relative risk 0.10–0.33), that greatly exceed those reported in meta-analyses of other trials. There were multiple examples of inconsistencies between and within trials, errors in reported data, misleading text, duplicated data and text, and uncertainties about ethical oversight.

Conclusions: A systematic approach using statistical techniques to assess randomization outcomes can evaluate data integrity, in this case suggesting these RCT results may be unreliable.

Original languageEnglish
Pages (from-to)2391-2402
Number of pages12
JournalNeurology
Volume87
Issue number23
Early online date9 Nov 2016
DOIs
Publication statusPublished - 6 Dec 2016

Fingerprint

Randomized Controlled Trials
Comorbidity
Research Personnel
Hip Fractures
Random Allocation
Uncertainty
Meta-Analysis
Confidence Intervals
Mortality
Incidence
Therapeutics

Keywords

  • osteoporosis
  • hip fracture
  • systematic review
  • statistical methods
  • research integrity

Cite this

Systematic review and statistical analysis of the integrity of 33 randomized controlled trials. / Bolland, Mark J; Avenell, Alison; Gamble, Greg D; Grey, Andrew.

In: Neurology, Vol. 87, No. 23, 06.12.2016, p. 2391-2402.

Research output: Contribution to journalArticle

Bolland, Mark J ; Avenell, Alison ; Gamble, Greg D ; Grey, Andrew. / Systematic review and statistical analysis of the integrity of 33 randomized controlled trials. In: Neurology. 2016 ; Vol. 87, No. 23. pp. 2391-2402.
@article{516202a1a5dc4c4f9ffeb0bd1f56c598,
title = "Systematic review and statistical analysis of the integrity of 33 randomized controlled trials",
abstract = "Background: Statistical techniques can investigate data integrity in randomized controlled trials (RCTs). We systematically reviewed and analyzed all human RCTs undertaken by a group of researchers, about which concerns have been raised.Methods: We compared observed distributions of p values for between-groups differences in baseline variables, for standardized sample means for continuous baseline variables, and for differences in treatment group participant numbers with the expected distributions. We assessed productivity, recruitment rates, outcome data, textual consistency, and ethical oversight.Results: The researchers were remarkably productive, publishing 33 RCTs over 15 years involving large numbers of older patients with substantial comorbidity, recruited over very short periods. Treatment groups were improbably similar. The distribution of p values for differences in baseline characteristics differed markedly from the expected uniform distribution (p = 5.2 × 10−82). The distribution of standardized sample means for baseline continuous variables and the differences between participant numbers in randomized groups also differed markedly from the expected distributions (p = 4.3 × 10−4, p = 1.5 × 10−5, respectively). Outcomes were remarkably positive, with very low mortality and study withdrawals despite substantial comorbidity. There were very large reductions in hip fracture incidence, regardless of intervention (relative risk 0.22, 95{\%} confidence interval 0.15–0.31, p < 0.0001, range of relative risk 0.10–0.33), that greatly exceed those reported in meta-analyses of other trials. There were multiple examples of inconsistencies between and within trials, errors in reported data, misleading text, duplicated data and text, and uncertainties about ethical oversight.Conclusions: A systematic approach using statistical techniques to assess randomization outcomes can evaluate data integrity, in this case suggesting these RCT results may be unreliable.",
keywords = "osteoporosis, hip fracture, systematic review, statistical methods, research integrity",
author = "Bolland, {Mark J} and Alison Avenell and Gamble, {Greg D} and Andrew Grey",
note = "Acknowledgements We are grateful to Bill Gillespie (Hull York Medical School, UK), Lesley Gillespie (University of Otago, New Zealand) and David Torgerson (University of York, UK) for earlier discussions with regard to these trials. We are also grateful for the assistance of Fiona Stewart, (Information Specialist) and Mari Imamura (Research Fellow and native Japanese speaker) from the University of Aberdeen. We thank Doug Altman and anonymous referees for their comments on previous drafts of this paper. Funding: Funded by the Health Research Council of New Zealand. The Health Services Research Unit is core funded by the Chief Scientist Office of the Scottish Government Health and Social Care Directorate.",
year = "2016",
month = "12",
day = "6",
doi = "10.1212/WNL.0000000000003387",
language = "English",
volume = "87",
pages = "2391--2402",
journal = "Neurology",
issn = "0028-3878",
publisher = "AAN Enterprises",
number = "23",

}

TY - JOUR

T1 - Systematic review and statistical analysis of the integrity of 33 randomized controlled trials

AU - Bolland, Mark J

AU - Avenell, Alison

AU - Gamble, Greg D

AU - Grey, Andrew

N1 - Acknowledgements We are grateful to Bill Gillespie (Hull York Medical School, UK), Lesley Gillespie (University of Otago, New Zealand) and David Torgerson (University of York, UK) for earlier discussions with regard to these trials. We are also grateful for the assistance of Fiona Stewart, (Information Specialist) and Mari Imamura (Research Fellow and native Japanese speaker) from the University of Aberdeen. We thank Doug Altman and anonymous referees for their comments on previous drafts of this paper. Funding: Funded by the Health Research Council of New Zealand. The Health Services Research Unit is core funded by the Chief Scientist Office of the Scottish Government Health and Social Care Directorate.

PY - 2016/12/6

Y1 - 2016/12/6

N2 - Background: Statistical techniques can investigate data integrity in randomized controlled trials (RCTs). We systematically reviewed and analyzed all human RCTs undertaken by a group of researchers, about which concerns have been raised.Methods: We compared observed distributions of p values for between-groups differences in baseline variables, for standardized sample means for continuous baseline variables, and for differences in treatment group participant numbers with the expected distributions. We assessed productivity, recruitment rates, outcome data, textual consistency, and ethical oversight.Results: The researchers were remarkably productive, publishing 33 RCTs over 15 years involving large numbers of older patients with substantial comorbidity, recruited over very short periods. Treatment groups were improbably similar. The distribution of p values for differences in baseline characteristics differed markedly from the expected uniform distribution (p = 5.2 × 10−82). The distribution of standardized sample means for baseline continuous variables and the differences between participant numbers in randomized groups also differed markedly from the expected distributions (p = 4.3 × 10−4, p = 1.5 × 10−5, respectively). Outcomes were remarkably positive, with very low mortality and study withdrawals despite substantial comorbidity. There were very large reductions in hip fracture incidence, regardless of intervention (relative risk 0.22, 95% confidence interval 0.15–0.31, p < 0.0001, range of relative risk 0.10–0.33), that greatly exceed those reported in meta-analyses of other trials. There were multiple examples of inconsistencies between and within trials, errors in reported data, misleading text, duplicated data and text, and uncertainties about ethical oversight.Conclusions: A systematic approach using statistical techniques to assess randomization outcomes can evaluate data integrity, in this case suggesting these RCT results may be unreliable.

AB - Background: Statistical techniques can investigate data integrity in randomized controlled trials (RCTs). We systematically reviewed and analyzed all human RCTs undertaken by a group of researchers, about which concerns have been raised.Methods: We compared observed distributions of p values for between-groups differences in baseline variables, for standardized sample means for continuous baseline variables, and for differences in treatment group participant numbers with the expected distributions. We assessed productivity, recruitment rates, outcome data, textual consistency, and ethical oversight.Results: The researchers were remarkably productive, publishing 33 RCTs over 15 years involving large numbers of older patients with substantial comorbidity, recruited over very short periods. Treatment groups were improbably similar. The distribution of p values for differences in baseline characteristics differed markedly from the expected uniform distribution (p = 5.2 × 10−82). The distribution of standardized sample means for baseline continuous variables and the differences between participant numbers in randomized groups also differed markedly from the expected distributions (p = 4.3 × 10−4, p = 1.5 × 10−5, respectively). Outcomes were remarkably positive, with very low mortality and study withdrawals despite substantial comorbidity. There were very large reductions in hip fracture incidence, regardless of intervention (relative risk 0.22, 95% confidence interval 0.15–0.31, p < 0.0001, range of relative risk 0.10–0.33), that greatly exceed those reported in meta-analyses of other trials. There were multiple examples of inconsistencies between and within trials, errors in reported data, misleading text, duplicated data and text, and uncertainties about ethical oversight.Conclusions: A systematic approach using statistical techniques to assess randomization outcomes can evaluate data integrity, in this case suggesting these RCT results may be unreliable.

KW - osteoporosis

KW - hip fracture

KW - systematic review

KW - statistical methods

KW - research integrity

U2 - 10.1212/WNL.0000000000003387

DO - 10.1212/WNL.0000000000003387

M3 - Article

VL - 87

SP - 2391

EP - 2402

JO - Neurology

JF - Neurology

SN - 0028-3878

IS - 23

ER -