How consistent are ratings and raters on the PCEPS? Interrater Reliability and Item Structure

Elizabeth Schmitt Freire, Robert Elliott, Graham Westwell

Research output: Contribution to conferencePaper

Abstract

Aim/Purpose: This study aimed to assess the reliability of the Person Centered and Experiential Psychotherapy Scale (PCEPS), a new adherence/competence measure of person-centred and experiential psychotherapies.

Design/Methodology: One-hundred twenty audio-recorded segments of therapy sessions selected from the archive of taped therapy sessions of the Strathclyde Therapy Research Centre were rated independently by two teams of three raters using the PCEPS. Six segments were systematically selected from 20 clients seen by 10 therapists (2 clients per therapist); in the first, middle, and last third of therapy; and in the first and second half of therapy sessions. Half of the segments were 10 min long and the other half were 15 min long. Six therapists were experienced therapists and four were counsellors in training. Seven of the therapists identified their work as 'person-centred’, and three of the experienced practitioners identifed their work as 'process-experiential' (although one of these was not fully trained in this approach). Three raters were qualified and experienced person-centred therapists and 3 raters were person-centred counselling trainees in their first year of training. In this paper we present the inter-item and inter-rater reliability results from this study, followed by exploratory factor analyses to study scale structure.

Results/Findings: Mean interrater alpha reliabilities for individual items varied from .68 (for Dominant or Overpowering Presence) to .86 (for Core Meaning). Average interrater reliability across the 15 items was .78, while the interrater reliability of the 15 items when averaged together was .87. Interitem reliabilities were quite high: .98 for the whole. Exploratory factor analyses revealed a 12-item facilitative relationship scale that cut across Person-centred and Experiential subscales (alpha: .98), and a nonfacilitative directiveness factor (3 items, alpha: .89).

Research Limitations: Therapy process measures in general and therapist adherence/competence measures in particular can be criticized for ignoring context and participant internal experiences, while attempting to generalize from relative brief segments of therapy to therapist performance in general. The present study would also be strengthened by comparison to non person-centred-experiential therapies.

Conclusions/Implications: The PCEPS has potential for use in RCT research as well as in counselling training and supervision, but will require further testing and validation.
Original languageEnglish
Publication statusUnpublished - 2012
Event10th Conference for the World Association for Person-Centered and Experiential Psychotherapy and Counseling - Antwerp, Belgium
Duration: 8 Jul 201212 Jul 2012

Conference

Conference10th Conference for the World Association for Person-Centered and Experiential Psychotherapy and Counseling
CountryBelgium
CityAntwerp
Period8/07/1212/07/12

Fingerprint

Psychotherapy
Mental Competency
Therapeutics
Statistical Factor Analysis
Counseling
Research
Process Assessment (Health Care)
Reproducibility of Results

Keywords

  • person-centred therapy
  • adherence/competence
  • experiential therapy
  • measure development

Cite this

Freire, E. S., Elliott, R., & Westwell, G. (2012). How consistent are ratings and raters on the PCEPS? Interrater Reliability and Item Structure. Paper presented at 10th Conference for the World Association for Person-Centered and Experiential Psychotherapy and Counseling, Antwerp, Belgium.

How consistent are ratings and raters on the PCEPS? Interrater Reliability and Item Structure. / Freire, Elizabeth Schmitt; Elliott, Robert; Westwell, Graham.

2012. Paper presented at 10th Conference for the World Association for Person-Centered and Experiential Psychotherapy and Counseling, Antwerp, Belgium.

Research output: Contribution to conferencePaper

Freire, ES, Elliott, R & Westwell, G 2012, 'How consistent are ratings and raters on the PCEPS? Interrater Reliability and Item Structure', Paper presented at 10th Conference for the World Association for Person-Centered and Experiential Psychotherapy and Counseling, Antwerp, Belgium, 8/07/12 - 12/07/12.
Freire ES, Elliott R, Westwell G. How consistent are ratings and raters on the PCEPS? Interrater Reliability and Item Structure. 2012. Paper presented at 10th Conference for the World Association for Person-Centered and Experiential Psychotherapy and Counseling, Antwerp, Belgium.
Freire, Elizabeth Schmitt ; Elliott, Robert ; Westwell, Graham. / How consistent are ratings and raters on the PCEPS? Interrater Reliability and Item Structure. Paper presented at 10th Conference for the World Association for Person-Centered and Experiential Psychotherapy and Counseling, Antwerp, Belgium.
@conference{5e4799b3818d479191f195059c0dbda7,
title = "How consistent are ratings and raters on the PCEPS?: Interrater Reliability and Item Structure",
abstract = "Aim/Purpose: This study aimed to assess the reliability of the Person Centered and Experiential Psychotherapy Scale (PCEPS), a new adherence/competence measure of person-centred and experiential psychotherapies.Design/Methodology: One-hundred twenty audio-recorded segments of therapy sessions selected from the archive of taped therapy sessions of the Strathclyde Therapy Research Centre were rated independently by two teams of three raters using the PCEPS. Six segments were systematically selected from 20 clients seen by 10 therapists (2 clients per therapist); in the first, middle, and last third of therapy; and in the first and second half of therapy sessions. Half of the segments were 10 min long and the other half were 15 min long. Six therapists were experienced therapists and four were counsellors in training. Seven of the therapists identified their work as 'person-centred’, and three of the experienced practitioners identifed their work as 'process-experiential' (although one of these was not fully trained in this approach). Three raters were qualified and experienced person-centred therapists and 3 raters were person-centred counselling trainees in their first year of training. In this paper we present the inter-item and inter-rater reliability results from this study, followed by exploratory factor analyses to study scale structure.Results/Findings: Mean interrater alpha reliabilities for individual items varied from .68 (for Dominant or Overpowering Presence) to .86 (for Core Meaning). Average interrater reliability across the 15 items was .78, while the interrater reliability of the 15 items when averaged together was .87. Interitem reliabilities were quite high: .98 for the whole. Exploratory factor analyses revealed a 12-item facilitative relationship scale that cut across Person-centred and Experiential subscales (alpha: .98), and a nonfacilitative directiveness factor (3 items, alpha: .89). Research Limitations: Therapy process measures in general and therapist adherence/competence measures in particular can be criticized for ignoring context and participant internal experiences, while attempting to generalize from relative brief segments of therapy to therapist performance in general. The present study would also be strengthened by comparison to non person-centred-experiential therapies.Conclusions/Implications: The PCEPS has potential for use in RCT research as well as in counselling training and supervision, but will require further testing and validation.",
keywords = "person-centred therapy, adherence/competence, experiential therapy, measure development",
author = "Freire, {Elizabeth Schmitt} and Robert Elliott and Graham Westwell",
year = "2012",
language = "English",
note = "10th Conference for the World Association for Person-Centered and Experiential Psychotherapy and Counseling ; Conference date: 08-07-2012 Through 12-07-2012",

}

TY - CONF

T1 - How consistent are ratings and raters on the PCEPS?

T2 - Interrater Reliability and Item Structure

AU - Freire, Elizabeth Schmitt

AU - Elliott, Robert

AU - Westwell, Graham

PY - 2012

Y1 - 2012

N2 - Aim/Purpose: This study aimed to assess the reliability of the Person Centered and Experiential Psychotherapy Scale (PCEPS), a new adherence/competence measure of person-centred and experiential psychotherapies.Design/Methodology: One-hundred twenty audio-recorded segments of therapy sessions selected from the archive of taped therapy sessions of the Strathclyde Therapy Research Centre were rated independently by two teams of three raters using the PCEPS. Six segments were systematically selected from 20 clients seen by 10 therapists (2 clients per therapist); in the first, middle, and last third of therapy; and in the first and second half of therapy sessions. Half of the segments were 10 min long and the other half were 15 min long. Six therapists were experienced therapists and four were counsellors in training. Seven of the therapists identified their work as 'person-centred’, and three of the experienced practitioners identifed their work as 'process-experiential' (although one of these was not fully trained in this approach). Three raters were qualified and experienced person-centred therapists and 3 raters were person-centred counselling trainees in their first year of training. In this paper we present the inter-item and inter-rater reliability results from this study, followed by exploratory factor analyses to study scale structure.Results/Findings: Mean interrater alpha reliabilities for individual items varied from .68 (for Dominant or Overpowering Presence) to .86 (for Core Meaning). Average interrater reliability across the 15 items was .78, while the interrater reliability of the 15 items when averaged together was .87. Interitem reliabilities were quite high: .98 for the whole. Exploratory factor analyses revealed a 12-item facilitative relationship scale that cut across Person-centred and Experiential subscales (alpha: .98), and a nonfacilitative directiveness factor (3 items, alpha: .89). Research Limitations: Therapy process measures in general and therapist adherence/competence measures in particular can be criticized for ignoring context and participant internal experiences, while attempting to generalize from relative brief segments of therapy to therapist performance in general. The present study would also be strengthened by comparison to non person-centred-experiential therapies.Conclusions/Implications: The PCEPS has potential for use in RCT research as well as in counselling training and supervision, but will require further testing and validation.

AB - Aim/Purpose: This study aimed to assess the reliability of the Person Centered and Experiential Psychotherapy Scale (PCEPS), a new adherence/competence measure of person-centred and experiential psychotherapies.Design/Methodology: One-hundred twenty audio-recorded segments of therapy sessions selected from the archive of taped therapy sessions of the Strathclyde Therapy Research Centre were rated independently by two teams of three raters using the PCEPS. Six segments were systematically selected from 20 clients seen by 10 therapists (2 clients per therapist); in the first, middle, and last third of therapy; and in the first and second half of therapy sessions. Half of the segments were 10 min long and the other half were 15 min long. Six therapists were experienced therapists and four were counsellors in training. Seven of the therapists identified their work as 'person-centred’, and three of the experienced practitioners identifed their work as 'process-experiential' (although one of these was not fully trained in this approach). Three raters were qualified and experienced person-centred therapists and 3 raters were person-centred counselling trainees in their first year of training. In this paper we present the inter-item and inter-rater reliability results from this study, followed by exploratory factor analyses to study scale structure.Results/Findings: Mean interrater alpha reliabilities for individual items varied from .68 (for Dominant or Overpowering Presence) to .86 (for Core Meaning). Average interrater reliability across the 15 items was .78, while the interrater reliability of the 15 items when averaged together was .87. Interitem reliabilities were quite high: .98 for the whole. Exploratory factor analyses revealed a 12-item facilitative relationship scale that cut across Person-centred and Experiential subscales (alpha: .98), and a nonfacilitative directiveness factor (3 items, alpha: .89). Research Limitations: Therapy process measures in general and therapist adherence/competence measures in particular can be criticized for ignoring context and participant internal experiences, while attempting to generalize from relative brief segments of therapy to therapist performance in general. The present study would also be strengthened by comparison to non person-centred-experiential therapies.Conclusions/Implications: The PCEPS has potential for use in RCT research as well as in counselling training and supervision, but will require further testing and validation.

KW - person-centred therapy

KW - adherence/competence

KW - experiential therapy

KW - measure development

M3 - Paper

ER -