The REal Life EVidence AssessmeNt Tool (RELEVANT): development of a novel quality assurance asset to rate observational comparative effectiveness research studies

Jonathan D Campbell (Corresponding Author), Robert Perry, Nikolaos G Papadopoulos, Jerry Krishnan, Guy Brusselle, Alison Chisholm, Leif Bjermer, Michael Thomas, Eric van Ganse, Maarten van den Berge, Jennifer Quint, David Price, Nicolas Roche

Research output: Contribution to journalArticle

1 Citation (Scopus)
3 Downloads (Pure)

Abstract

Background: Evidence from observational comparative effectiveness research (CER) is ranked below that from randomized controlled trials in traditional evidence hierarchies. However, asthma observational CER studies represent an important complementary evidence source answering different research questions and are particularly valuable in guiding clinical decision making in real-life patient and practice settings. Tools are required to assist in quality appraisal of observational CER to enable identification of and confidence in high-quality CER evidence to inform guideline development.

Methods: The REal Life EVidence AssessmeNt Tool (RELEVANT) was developed through a step-wise approach. We conducted an iterative refinement of the tool based on Task Force member expertise and feedback from pilot testing the tool until reaching adequate inter-rater agreement percentages. Two distinct pilots were conducted-the first involving six members of the Respiratory Effectiveness Group (REG) and European Academy of Allergy and Clinical Immunology (EAACI) joint Task Force for quality appraisal of observational asthma CER; the second involving 22 members of REG and EAACI membership. The final tool consists of 21 quality sub-items distributed across seven methodology domains: Background, Design, Measures, Analysis, Results, Discussion/Interpretation, and Conflict of Interest. Eleven of these sub-items are considered critical and named "primary sub-items".

Results: Following the second pilot, RELEVANT showed inter-rater agreement ≥ 70% for 94% of all primary and 93% for all secondary sub-items tested across three rater groups. For observational CER to be classified as sufficiently high quality for future guideline consideration, all RELEVANT primary sub-items must be fulfilled. The ten secondary sub-items further qualify the relative strengths and weaknesses of the published CER evidence. RELEVANT could also be applicable to general quality appraisal of observational CER across other medical specialties.

Conclusions: RELEVANT is the first quality checklist to assist in the appraisal of published observational CER developed through iterative feedback derived from pilot implementation and inter-rater agreement evaluation. Developed for a REG-EAACI Task Force quality appraisal of recent asthma CER, RELEVANT also has wider utility to support appraisal of CER literature in general (including pre-publication). It may also assist in manuscript development and in educating relevant stakeholders about key quality markers in observational CER.

Original languageEnglish
Article number21
Number of pages11
JournalClinical and Translational Allergy
Volume9
Issue number21
DOIs
Publication statusPublished - 27 Mar 2019

Fingerprint

Comparative Effectiveness Research
Advisory Committees
Allergy and Immunology
Asthma
Guidelines
Conflict of Interest
Manuscripts
Checklist

Keywords

  • Asthma
  • Comparative effectiveness research (CER)
  • Quality
  • Observational studies
  • Assessment tool
  • RELEVANT
  • BECLOMETASONE
  • POPULATION
  • DATABASE
  • ASTHMA
  • WORLD RESEARCH

ASJC Scopus subject areas

  • Pulmonary and Respiratory Medicine
  • Immunology and Allergy
  • Immunology

Cite this

The REal Life EVidence AssessmeNt Tool (RELEVANT) : development of a novel quality assurance asset to rate observational comparative effectiveness research studies. / Campbell, Jonathan D (Corresponding Author); Perry, Robert; Papadopoulos, Nikolaos G; Krishnan, Jerry; Brusselle, Guy; Chisholm, Alison; Bjermer, Leif; Thomas, Michael; van Ganse, Eric; van den Berge, Maarten; Quint, Jennifer; Price, David; Roche, Nicolas.

In: Clinical and Translational Allergy, Vol. 9, No. 21, 21, 27.03.2019.

Research output: Contribution to journalArticle

Campbell, JD, Perry, R, Papadopoulos, NG, Krishnan, J, Brusselle, G, Chisholm, A, Bjermer, L, Thomas, M, van Ganse, E, van den Berge, M, Quint, J, Price, D & Roche, N 2019, 'The REal Life EVidence AssessmeNt Tool (RELEVANT): development of a novel quality assurance asset to rate observational comparative effectiveness research studies', Clinical and Translational Allergy, vol. 9, no. 21, 21. https://doi.org/10.1186/s13601-019-0256-9
Campbell, Jonathan D ; Perry, Robert ; Papadopoulos, Nikolaos G ; Krishnan, Jerry ; Brusselle, Guy ; Chisholm, Alison ; Bjermer, Leif ; Thomas, Michael ; van Ganse, Eric ; van den Berge, Maarten ; Quint, Jennifer ; Price, David ; Roche, Nicolas. / The REal Life EVidence AssessmeNt Tool (RELEVANT) : development of a novel quality assurance asset to rate observational comparative effectiveness research studies. In: Clinical and Translational Allergy. 2019 ; Vol. 9, No. 21.
@article{b69aebefea9448e4a497b2e95919c032,
title = "The REal Life EVidence AssessmeNt Tool (RELEVANT): development of a novel quality assurance asset to rate observational comparative effectiveness research studies",
abstract = "Background: Evidence from observational comparative effectiveness research (CER) is ranked below that from randomized controlled trials in traditional evidence hierarchies. However, asthma observational CER studies represent an important complementary evidence source answering different research questions and are particularly valuable in guiding clinical decision making in real-life patient and practice settings. Tools are required to assist in quality appraisal of observational CER to enable identification of and confidence in high-quality CER evidence to inform guideline development.Methods: The REal Life EVidence AssessmeNt Tool (RELEVANT) was developed through a step-wise approach. We conducted an iterative refinement of the tool based on Task Force member expertise and feedback from pilot testing the tool until reaching adequate inter-rater agreement percentages. Two distinct pilots were conducted-the first involving six members of the Respiratory Effectiveness Group (REG) and European Academy of Allergy and Clinical Immunology (EAACI) joint Task Force for quality appraisal of observational asthma CER; the second involving 22 members of REG and EAACI membership. The final tool consists of 21 quality sub-items distributed across seven methodology domains: Background, Design, Measures, Analysis, Results, Discussion/Interpretation, and Conflict of Interest. Eleven of these sub-items are considered critical and named {"}primary sub-items{"}.Results: Following the second pilot, RELEVANT showed inter-rater agreement ≥ 70{\%} for 94{\%} of all primary and 93{\%} for all secondary sub-items tested across three rater groups. For observational CER to be classified as sufficiently high quality for future guideline consideration, all RELEVANT primary sub-items must be fulfilled. The ten secondary sub-items further qualify the relative strengths and weaknesses of the published CER evidence. RELEVANT could also be applicable to general quality appraisal of observational CER across other medical specialties.Conclusions: RELEVANT is the first quality checklist to assist in the appraisal of published observational CER developed through iterative feedback derived from pilot implementation and inter-rater agreement evaluation. Developed for a REG-EAACI Task Force quality appraisal of recent asthma CER, RELEVANT also has wider utility to support appraisal of CER literature in general (including pre-publication). It may also assist in manuscript development and in educating relevant stakeholders about key quality markers in observational CER.",
keywords = "Asthma, Comparative effectiveness research (CER), Quality, Observational studies, Assessment tool, RELEVANT, BECLOMETASONE, POPULATION, DATABASE, ASTHMA, WORLD RESEARCH",
author = "Campbell, {Jonathan D} and Robert Perry and Papadopoulos, {Nikolaos G} and Jerry Krishnan and Guy Brusselle and Alison Chisholm and Leif Bjermer and Michael Thomas and {van Ganse}, Eric and {van den Berge}, Maarten and Jennifer Quint and David Price and Nicolas Roche",
note = "Funding for this research was provided by: European Academy of Allergy and Clinical Immunology",
year = "2019",
month = "3",
day = "27",
doi = "10.1186/s13601-019-0256-9",
language = "English",
volume = "9",
journal = "Clinical and Translational Allergy",
issn = "2045-7022",
publisher = "BioMed Central",
number = "21",

}

TY - JOUR

T1 - The REal Life EVidence AssessmeNt Tool (RELEVANT)

T2 - development of a novel quality assurance asset to rate observational comparative effectiveness research studies

AU - Campbell, Jonathan D

AU - Perry, Robert

AU - Papadopoulos, Nikolaos G

AU - Krishnan, Jerry

AU - Brusselle, Guy

AU - Chisholm, Alison

AU - Bjermer, Leif

AU - Thomas, Michael

AU - van Ganse, Eric

AU - van den Berge, Maarten

AU - Quint, Jennifer

AU - Price, David

AU - Roche, Nicolas

N1 - Funding for this research was provided by: European Academy of Allergy and Clinical Immunology

PY - 2019/3/27

Y1 - 2019/3/27

N2 - Background: Evidence from observational comparative effectiveness research (CER) is ranked below that from randomized controlled trials in traditional evidence hierarchies. However, asthma observational CER studies represent an important complementary evidence source answering different research questions and are particularly valuable in guiding clinical decision making in real-life patient and practice settings. Tools are required to assist in quality appraisal of observational CER to enable identification of and confidence in high-quality CER evidence to inform guideline development.Methods: The REal Life EVidence AssessmeNt Tool (RELEVANT) was developed through a step-wise approach. We conducted an iterative refinement of the tool based on Task Force member expertise and feedback from pilot testing the tool until reaching adequate inter-rater agreement percentages. Two distinct pilots were conducted-the first involving six members of the Respiratory Effectiveness Group (REG) and European Academy of Allergy and Clinical Immunology (EAACI) joint Task Force for quality appraisal of observational asthma CER; the second involving 22 members of REG and EAACI membership. The final tool consists of 21 quality sub-items distributed across seven methodology domains: Background, Design, Measures, Analysis, Results, Discussion/Interpretation, and Conflict of Interest. Eleven of these sub-items are considered critical and named "primary sub-items".Results: Following the second pilot, RELEVANT showed inter-rater agreement ≥ 70% for 94% of all primary and 93% for all secondary sub-items tested across three rater groups. For observational CER to be classified as sufficiently high quality for future guideline consideration, all RELEVANT primary sub-items must be fulfilled. The ten secondary sub-items further qualify the relative strengths and weaknesses of the published CER evidence. RELEVANT could also be applicable to general quality appraisal of observational CER across other medical specialties.Conclusions: RELEVANT is the first quality checklist to assist in the appraisal of published observational CER developed through iterative feedback derived from pilot implementation and inter-rater agreement evaluation. Developed for a REG-EAACI Task Force quality appraisal of recent asthma CER, RELEVANT also has wider utility to support appraisal of CER literature in general (including pre-publication). It may also assist in manuscript development and in educating relevant stakeholders about key quality markers in observational CER.

AB - Background: Evidence from observational comparative effectiveness research (CER) is ranked below that from randomized controlled trials in traditional evidence hierarchies. However, asthma observational CER studies represent an important complementary evidence source answering different research questions and are particularly valuable in guiding clinical decision making in real-life patient and practice settings. Tools are required to assist in quality appraisal of observational CER to enable identification of and confidence in high-quality CER evidence to inform guideline development.Methods: The REal Life EVidence AssessmeNt Tool (RELEVANT) was developed through a step-wise approach. We conducted an iterative refinement of the tool based on Task Force member expertise and feedback from pilot testing the tool until reaching adequate inter-rater agreement percentages. Two distinct pilots were conducted-the first involving six members of the Respiratory Effectiveness Group (REG) and European Academy of Allergy and Clinical Immunology (EAACI) joint Task Force for quality appraisal of observational asthma CER; the second involving 22 members of REG and EAACI membership. The final tool consists of 21 quality sub-items distributed across seven methodology domains: Background, Design, Measures, Analysis, Results, Discussion/Interpretation, and Conflict of Interest. Eleven of these sub-items are considered critical and named "primary sub-items".Results: Following the second pilot, RELEVANT showed inter-rater agreement ≥ 70% for 94% of all primary and 93% for all secondary sub-items tested across three rater groups. For observational CER to be classified as sufficiently high quality for future guideline consideration, all RELEVANT primary sub-items must be fulfilled. The ten secondary sub-items further qualify the relative strengths and weaknesses of the published CER evidence. RELEVANT could also be applicable to general quality appraisal of observational CER across other medical specialties.Conclusions: RELEVANT is the first quality checklist to assist in the appraisal of published observational CER developed through iterative feedback derived from pilot implementation and inter-rater agreement evaluation. Developed for a REG-EAACI Task Force quality appraisal of recent asthma CER, RELEVANT also has wider utility to support appraisal of CER literature in general (including pre-publication). It may also assist in manuscript development and in educating relevant stakeholders about key quality markers in observational CER.

KW - Asthma

KW - Comparative effectiveness research (CER)

KW - Quality

KW - Observational studies

KW - Assessment tool

KW - RELEVANT

KW - BECLOMETASONE

KW - POPULATION

KW - DATABASE

KW - ASTHMA

KW - WORLD RESEARCH

UR - http://www.scopus.com/inward/record.url?scp=85063421513&partnerID=8YFLogxK

UR - http://www.mendeley.com/research/real-life-evidence-assessment-tool-relevant-development-novel-quality-assurance-asset-rate-observati

U2 - 10.1186/s13601-019-0256-9

DO - 10.1186/s13601-019-0256-9

M3 - Article

C2 - 30962876

VL - 9

JO - Clinical and Translational Allergy

JF - Clinical and Translational Allergy

SN - 2045-7022

IS - 21

M1 - 21

ER -