Rating of Included Trials on the Efficacy-Effectiveness Spectrum: development of a new tool for systematic reviews

L. Susan  Wieland; Brian M.  Berman; Douglas G. Altman; Jürgen  Barth; Lex M. Bouter; Christopher R.  D’Adamo; Klaus  Linde; David Moher; Daniel  Mullins; Shaun Treweek; Sean Tunis; Danielle A.  van der Windt; Merrick Zwarenstein; Claudia Witt

doi:10.1016/j.jclinepi.2017.01.010

Rating of Included Trials on the Efficacy-Effectiveness Spectrum: development of a new tool for systematic reviews

L. Susan Wieland, Brian M. Berman, Douglas G. Altman, Jürgen Barth, Lex M. Bouter, Christopher R. D’Adamo, Klaus Linde, David Moher, Daniel Mullins, Shaun Treweek, Sean Tunis, Danielle A. van der Windt, Merrick Zwarenstein, Claudia Witt

Research output: Contribution to journal › Article › peer-review

30 Citations (Scopus)

9 Downloads (Pure)

Abstract

Background: Randomized trials may be designed to provide evidence more related to intervention efficacy or effectiveness. When systematic reviews are used to inform clinical or policy decisions, it is important to know the efficacy-effectiveness nature of information from the included trials. Objective: To develop a tool to characterize randomized trials information in a systematic review on an efficacy-effectiveness continuum. Methods: We extracted rating domains and descriptors from existing tools, and used a modified Delphi procedure to condense domains and develop a new tool. The feasibility and inter-rater reliability of the tool was tested on trials from 4 systematic reviews. Results: The RITES (Rating of Included Trials on the Efficacy-effectiveness Spectrum) tool rates clinical trials on a 5-point Likert scale in four domains: (1) participant characteristics, (2) trial setting, (3) flexibility of interventions, (4) clinical relevance of interventions. When RITES was piloted on trials from 3 reviews by unaffiliated raters, ratings were variable (Intraclass Correlation Coefficients (ICCs) 0.23-0.45 for the four domains; 0.25-0.66 after excluding outliers), but when used on one review by the review authors with expertise on the topic the ratings were consistent (ICCs >0.80). Conclusion: RITES may help characterize the efficacy-effectiveness nature of trial information in systematic reviews.

Original language	English
Pages (from-to)	95-104
Number of pages	10
Journal	Journal of Clinical Epidemiology
Volume	84
Early online date	7 Feb 2017
DOIs	https://doi.org/10.1016/j.jclinepi.2017.01.010
Publication status	Published - Apr 2017

Bibliographical note

Funding: This work was supported by the National Institutes of Health National Center for Complementary and Integrative Health (R24 AT001293)

Keywords

Comparative Effectiveness Research
Systematic reviews
Randomized controlled trials
Effectiveness
Efficacy

Access to Document

10.1016/j.jclinepi.2017.01.010Licence: Unspecified

Accepted Manuscript
© 2017. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/
Accepted author manuscript, 972 KBLicence: CC BY-NC-ND

Cite this

Wieland, L. S., Berman, B. M., Altman, D. G., Barth, J., Bouter, L. M., D’Adamo, C. R., Linde, K., Moher, D., Mullins, D., Treweek, S., Tunis, S., van der Windt, D. A., Zwarenstein, M., & Witt, C. (2017). Rating of Included Trials on the Efficacy-Effectiveness Spectrum: development of a new tool for systematic reviews. Journal of Clinical Epidemiology, 84, 95-104. https://doi.org/10.1016/j.jclinepi.2017.01.010

Wieland, LS, Berman, BM, Altman, DG, Barth, J, Bouter, LM, D’Adamo, CR, Linde, K, Moher, D, Mullins, D, Treweek, S, Tunis, S, van der Windt, DA, Zwarenstein, M & Witt, C 2017, 'Rating of Included Trials on the Efficacy-Effectiveness Spectrum: development of a new tool for systematic reviews', Journal of Clinical Epidemiology, vol. 84, pp. 95-104. https://doi.org/10.1016/j.jclinepi.2017.01.010

@article{157d82dc1cba471dacf7c92a4917485c,

title = "Rating of Included Trials on the Efficacy-Effectiveness Spectrum: development of a new tool for systematic reviews",

abstract = "Background: Randomized trials may be designed to provide evidence more related to intervention efficacy or effectiveness. When systematic reviews are used to inform clinical or policy decisions, it is important to know the efficacy-effectiveness nature of information from the included trials. Objective: To develop a tool to characterize randomized trials information in a systematic review on an efficacy-effectiveness continuum. Methods: We extracted rating domains and descriptors from existing tools, and used a modified Delphi procedure to condense domains and develop a new tool. The feasibility and inter-rater reliability of the tool was tested on trials from 4 systematic reviews. Results: The RITES (Rating of Included Trials on the Efficacy-effectiveness Spectrum) tool rates clinical trials on a 5-point Likert scale in four domains: (1) participant characteristics, (2) trial setting, (3) flexibility of interventions, (4) clinical relevance of interventions. When RITES was piloted on trials from 3 reviews by unaffiliated raters, ratings were variable (Intraclass Correlation Coefficients (ICCs) 0.23-0.45 for the four domains; 0.25-0.66 after excluding outliers), but when used on one review by the review authors with expertise on the topic the ratings were consistent (ICCs >0.80). Conclusion: RITES may help characterize the efficacy-effectiveness nature of trial information in systematic reviews. ",

keywords = "Comparative Effectiveness Research, Systematic reviews, Randomized controlled trials, Effectiveness, Efficacy",

author = "Wieland, {L. Susan} and Berman, {Brian M.} and Altman, {Douglas G.} and J{\"u}rgen Barth and Bouter, {Lex M.} and D{\textquoteright}Adamo, {Christopher R.} and Klaus Linde and David Moher and Daniel Mullins and Shaun Treweek and Sean Tunis and {van der Windt}, {Danielle A.} and Merrick Zwarenstein and Claudia Witt",

note = "Funding: This work was supported by the National Institutes of Health National Center for Complementary and Integrative Health (R24 AT001293)",

year = "2017",

month = apr,

doi = "10.1016/j.jclinepi.2017.01.010",

language = "English",

volume = "84",

pages = "95--104",

journal = "Journal of Clinical Epidemiology",

issn = "0895-4356",

publisher = "Elsevier USA",

}

TY - JOUR

T1 - Rating of Included Trials on the Efficacy-Effectiveness Spectrum

T2 - development of a new tool for systematic reviews

AU - Wieland, L. Susan

AU - Berman, Brian M.

AU - Altman, Douglas G.

AU - Barth, Jürgen

AU - Bouter, Lex M.

AU - D’Adamo, Christopher R.

AU - Linde, Klaus

AU - Moher, David

AU - Mullins, Daniel

AU - Treweek, Shaun

AU - Tunis, Sean

AU - van der Windt, Danielle A.

AU - Zwarenstein, Merrick

AU - Witt, Claudia

N1 - Funding: This work was supported by the National Institutes of Health National Center for Complementary and Integrative Health (R24 AT001293)

PY - 2017/4

Y1 - 2017/4

N2 - Background: Randomized trials may be designed to provide evidence more related to intervention efficacy or effectiveness. When systematic reviews are used to inform clinical or policy decisions, it is important to know the efficacy-effectiveness nature of information from the included trials. Objective: To develop a tool to characterize randomized trials information in a systematic review on an efficacy-effectiveness continuum. Methods: We extracted rating domains and descriptors from existing tools, and used a modified Delphi procedure to condense domains and develop a new tool. The feasibility and inter-rater reliability of the tool was tested on trials from 4 systematic reviews. Results: The RITES (Rating of Included Trials on the Efficacy-effectiveness Spectrum) tool rates clinical trials on a 5-point Likert scale in four domains: (1) participant characteristics, (2) trial setting, (3) flexibility of interventions, (4) clinical relevance of interventions. When RITES was piloted on trials from 3 reviews by unaffiliated raters, ratings were variable (Intraclass Correlation Coefficients (ICCs) 0.23-0.45 for the four domains; 0.25-0.66 after excluding outliers), but when used on one review by the review authors with expertise on the topic the ratings were consistent (ICCs >0.80). Conclusion: RITES may help characterize the efficacy-effectiveness nature of trial information in systematic reviews.

AB - Background: Randomized trials may be designed to provide evidence more related to intervention efficacy or effectiveness. When systematic reviews are used to inform clinical or policy decisions, it is important to know the efficacy-effectiveness nature of information from the included trials. Objective: To develop a tool to characterize randomized trials information in a systematic review on an efficacy-effectiveness continuum. Methods: We extracted rating domains and descriptors from existing tools, and used a modified Delphi procedure to condense domains and develop a new tool. The feasibility and inter-rater reliability of the tool was tested on trials from 4 systematic reviews. Results: The RITES (Rating of Included Trials on the Efficacy-effectiveness Spectrum) tool rates clinical trials on a 5-point Likert scale in four domains: (1) participant characteristics, (2) trial setting, (3) flexibility of interventions, (4) clinical relevance of interventions. When RITES was piloted on trials from 3 reviews by unaffiliated raters, ratings were variable (Intraclass Correlation Coefficients (ICCs) 0.23-0.45 for the four domains; 0.25-0.66 after excluding outliers), but when used on one review by the review authors with expertise on the topic the ratings were consistent (ICCs >0.80). Conclusion: RITES may help characterize the efficacy-effectiveness nature of trial information in systematic reviews.

KW - Comparative Effectiveness Research

KW - Systematic reviews

KW - Randomized controlled trials

KW - Effectiveness

KW - Efficacy

U2 - 10.1016/j.jclinepi.2017.01.010

DO - 10.1016/j.jclinepi.2017.01.010

M3 - Article

SN - 0895-4356

VL - 84

SP - 95

EP - 104

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

ER -

Rating of Included Trials on the Efficacy-Effectiveness Spectrum: development of a new tool for systematic reviews

Abstract

Bibliographical note

Keywords

Access to Document

Fingerprint

Cite this