Systematic review and statistical analysis of the integrity of 33 randomized controlled trials

Mark J Bolland; Alison Avenell; Greg D Gamble; Andrew Grey

doi:10.1212/WNL.0000000000003387

Systematic review and statistical analysis of the integrity of 33 randomized controlled trials

Mark J Bolland, Alison Avenell, Greg D Gamble, Andrew Grey

The University of Auckland

Research output: Contribution to journal › Article › peer-review

67 Citations (Scopus)

Abstract

Background: Statistical techniques can investigate data integrity in randomized controlled trials (RCTs). We systematically reviewed and analyzed all human RCTs undertaken by a group of researchers, about which concerns have been raised.

Methods: We compared observed distributions of p values for between-groups differences in baseline variables, for standardized sample means for continuous baseline variables, and for differences in treatment group participant numbers with the expected distributions. We assessed productivity, recruitment rates, outcome data, textual consistency, and ethical oversight.

Results: The researchers were remarkably productive, publishing 33 RCTs over 15 years involving large numbers of older patients with substantial comorbidity, recruited over very short periods. Treatment groups were improbably similar. The distribution of p values for differences in baseline characteristics differed markedly from the expected uniform distribution (p = 5.2 × 10⁻⁸²). The distribution of standardized sample means for baseline continuous variables and the differences between participant numbers in randomized groups also differed markedly from the expected distributions (p = 4.3 × 10⁻⁴, p = 1.5 × 10⁻⁵, respectively). Outcomes were remarkably positive, with very low mortality and study withdrawals despite substantial comorbidity. There were very large reductions in hip fracture incidence, regardless of intervention (relative risk 0.22, 95% confidence interval 0.15–0.31, p < 0.0001, range of relative risk 0.10–0.33), that greatly exceed those reported in meta-analyses of other trials. There were multiple examples of inconsistencies between and within trials, errors in reported data, misleading text, duplicated data and text, and uncertainties about ethical oversight.

Conclusions: A systematic approach using statistical techniques to assess randomization outcomes can evaluate data integrity, in this case suggesting these RCT results may be unreliable.

Original language	English
Pages (from-to)	2391-2402
Number of pages	12
Journal	Neurology
Volume	87
Issue number	23
Early online date	9 Nov 2016
DOIs	https://doi.org/10.1212/WNL.0000000000003387
Publication status	Published - 6 Dec 2016

Bibliographical note

Acknowledgements
We are grateful to Bill Gillespie (Hull York Medical School, UK), Lesley Gillespie (University of Otago, New Zealand) and David Torgerson (University of York, UK) for earlier discussions with regard to these trials. We are also grateful for the assistance of Fiona Stewart, (Information Specialist) and Mari Imamura (Research Fellow and native Japanese speaker) from the University of Aberdeen. We thank Doug Altman and anonymous referees for their comments on previous drafts of this paper.

Funding:
Funded by the Health Research Council of New Zealand. The Health Services Research Unit is core funded by the Chief Scientist Office of the Scottish Government Health and Social Care Directorate.

Keywords

osteoporosis
hip fracture
systematic review
statistical methods
research integrity

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1212/WNL.0000000000003387Licence: Unspecified

Alison Avenell
- School of Medicine, Medical Sciences & Nutrition, Health Services Research Unit (HSRU) - Clinical Chair in Health Services Research
- Institute of Applied Health Sciences
Person: Clinical Academic

Cite this

@article{516202a1a5dc4c4f9ffeb0bd1f56c598,

title = "Systematic review and statistical analysis of the integrity of 33 randomized controlled trials",

abstract = "Background: Statistical techniques can investigate data integrity in randomized controlled trials (RCTs). We systematically reviewed and analyzed all human RCTs undertaken by a group of researchers, about which concerns have been raised.Methods: We compared observed distributions of p values for between-groups differences in baseline variables, for standardized sample means for continuous baseline variables, and for differences in treatment group participant numbers with the expected distributions. We assessed productivity, recruitment rates, outcome data, textual consistency, and ethical oversight.Results: The researchers were remarkably productive, publishing 33 RCTs over 15 years involving large numbers of older patients with substantial comorbidity, recruited over very short periods. Treatment groups were improbably similar. The distribution of p values for differences in baseline characteristics differed markedly from the expected uniform distribution (p = 5.2 × 10−82). The distribution of standardized sample means for baseline continuous variables and the differences between participant numbers in randomized groups also differed markedly from the expected distributions (p = 4.3 × 10−4, p = 1.5 × 10−5, respectively). Outcomes were remarkably positive, with very low mortality and study withdrawals despite substantial comorbidity. There were very large reductions in hip fracture incidence, regardless of intervention (relative risk 0.22, 95% confidence interval 0.15–0.31, p < 0.0001, range of relative risk 0.10–0.33), that greatly exceed those reported in meta-analyses of other trials. There were multiple examples of inconsistencies between and within trials, errors in reported data, misleading text, duplicated data and text, and uncertainties about ethical oversight.Conclusions: A systematic approach using statistical techniques to assess randomization outcomes can evaluate data integrity, in this case suggesting these RCT results may be unreliable.",

keywords = "osteoporosis, hip fracture, systematic review, statistical methods, research integrity",

author = "Bolland, {Mark J} and Alison Avenell and Gamble, {Greg D} and Andrew Grey",

note = "Acknowledgements We are grateful to Bill Gillespie (Hull York Medical School, UK), Lesley Gillespie (University of Otago, New Zealand) and David Torgerson (University of York, UK) for earlier discussions with regard to these trials. We are also grateful for the assistance of Fiona Stewart, (Information Specialist) and Mari Imamura (Research Fellow and native Japanese speaker) from the University of Aberdeen. We thank Doug Altman and anonymous referees for their comments on previous drafts of this paper. Funding: Funded by the Health Research Council of New Zealand. The Health Services Research Unit is core funded by the Chief Scientist Office of the Scottish Government Health and Social Care Directorate. ",

year = "2016",

month = dec,

day = "6",

doi = "10.1212/WNL.0000000000003387",

language = "English",

volume = "87",

pages = "2391--2402",

journal = "Neurology",

issn = "0028-3878",

publisher = "AAN Enterprises",

number = "23",

}

TY - JOUR

T1 - Systematic review and statistical analysis of the integrity of 33 randomized controlled trials

AU - Bolland, Mark J

AU - Avenell, Alison

AU - Gamble, Greg D

AU - Grey, Andrew

N1 - Acknowledgements We are grateful to Bill Gillespie (Hull York Medical School, UK), Lesley Gillespie (University of Otago, New Zealand) and David Torgerson (University of York, UK) for earlier discussions with regard to these trials. We are also grateful for the assistance of Fiona Stewart, (Information Specialist) and Mari Imamura (Research Fellow and native Japanese speaker) from the University of Aberdeen. We thank Doug Altman and anonymous referees for their comments on previous drafts of this paper. Funding: Funded by the Health Research Council of New Zealand. The Health Services Research Unit is core funded by the Chief Scientist Office of the Scottish Government Health and Social Care Directorate.

PY - 2016/12/6

Y1 - 2016/12/6

N2 - Background: Statistical techniques can investigate data integrity in randomized controlled trials (RCTs). We systematically reviewed and analyzed all human RCTs undertaken by a group of researchers, about which concerns have been raised.Methods: We compared observed distributions of p values for between-groups differences in baseline variables, for standardized sample means for continuous baseline variables, and for differences in treatment group participant numbers with the expected distributions. We assessed productivity, recruitment rates, outcome data, textual consistency, and ethical oversight.Results: The researchers were remarkably productive, publishing 33 RCTs over 15 years involving large numbers of older patients with substantial comorbidity, recruited over very short periods. Treatment groups were improbably similar. The distribution of p values for differences in baseline characteristics differed markedly from the expected uniform distribution (p = 5.2 × 10−82). The distribution of standardized sample means for baseline continuous variables and the differences between participant numbers in randomized groups also differed markedly from the expected distributions (p = 4.3 × 10−4, p = 1.5 × 10−5, respectively). Outcomes were remarkably positive, with very low mortality and study withdrawals despite substantial comorbidity. There were very large reductions in hip fracture incidence, regardless of intervention (relative risk 0.22, 95% confidence interval 0.15–0.31, p < 0.0001, range of relative risk 0.10–0.33), that greatly exceed those reported in meta-analyses of other trials. There were multiple examples of inconsistencies between and within trials, errors in reported data, misleading text, duplicated data and text, and uncertainties about ethical oversight.Conclusions: A systematic approach using statistical techniques to assess randomization outcomes can evaluate data integrity, in this case suggesting these RCT results may be unreliable.

AB - Background: Statistical techniques can investigate data integrity in randomized controlled trials (RCTs). We systematically reviewed and analyzed all human RCTs undertaken by a group of researchers, about which concerns have been raised.Methods: We compared observed distributions of p values for between-groups differences in baseline variables, for standardized sample means for continuous baseline variables, and for differences in treatment group participant numbers with the expected distributions. We assessed productivity, recruitment rates, outcome data, textual consistency, and ethical oversight.Results: The researchers were remarkably productive, publishing 33 RCTs over 15 years involving large numbers of older patients with substantial comorbidity, recruited over very short periods. Treatment groups were improbably similar. The distribution of p values for differences in baseline characteristics differed markedly from the expected uniform distribution (p = 5.2 × 10−82). The distribution of standardized sample means for baseline continuous variables and the differences between participant numbers in randomized groups also differed markedly from the expected distributions (p = 4.3 × 10−4, p = 1.5 × 10−5, respectively). Outcomes were remarkably positive, with very low mortality and study withdrawals despite substantial comorbidity. There were very large reductions in hip fracture incidence, regardless of intervention (relative risk 0.22, 95% confidence interval 0.15–0.31, p < 0.0001, range of relative risk 0.10–0.33), that greatly exceed those reported in meta-analyses of other trials. There were multiple examples of inconsistencies between and within trials, errors in reported data, misleading text, duplicated data and text, and uncertainties about ethical oversight.Conclusions: A systematic approach using statistical techniques to assess randomization outcomes can evaluate data integrity, in this case suggesting these RCT results may be unreliable.

KW - osteoporosis

KW - hip fracture

KW - systematic review

KW - statistical methods

KW - research integrity

U2 - 10.1212/WNL.0000000000003387

DO - 10.1212/WNL.0000000000003387

M3 - Article

SN - 0028-3878

VL - 87

SP - 2391

EP - 2402

JO - Neurology

JF - Neurology

IS - 23

ER -

Systematic review and statistical analysis of the integrity of 33 randomized controlled trials

Abstract

Bibliographical note

Keywords

UN SDGs

Access to Document

Fingerprint

Profiles

Alison Avenell

Cite this