Methods of testing for a deficit in single-case studies: Evaluation of statistical power by Monte Carlo simulation

John Robertson Crawford, P. H. Garthwaite

Research output: Contribution to journalArticle

49 Citations (Scopus)

Abstract

Testing for the presence of a deficit by comparing a case to controls is a fundamental feature of many neuropsychological single-case studies. Monte Carlo simulation was employed to study the statistical power of two competing approaches to this task. The power to detect a large deficit was low to moderate for a method proposed by Crawford and Howell (1998; ranging from 44% to 63%) but was extremely low for a method proposed by Mycroft, Mitchell, and Kay (2002; ranging from 1% to 13%). The effects of departures from normality were examined, as was the effect of varying degrees of measurement error in the scores of controls and the single case. Measurement error produced a moderate reduction in power when present in both controls and the case; the effect of differentially greater measurement error for the single case depended on the initial level of power. When Mycroft et al.'s method was used to test for the presence of a classical dissociation, it produced very high Type I error rates (ranging from 20.7% to 49.3%); in contrast, the rates for criteria proposed by Crawford and Garthwaite (2005b) were low (ranging from 1.3% to 6.7%). The broader implications of these results for single-case research are discussed.

Original languageEnglish
Pages (from-to)877-904
Number of pages27
JournalCognitive Neuropsychology
Volume23
Issue number6
DOIs
Publication statusPublished - Sep 2006

Keywords

  • SKEW-NORMAL-DISTRIBUTION
  • T-DISTRIBUTION
  • ALZHEIMERS-DISEASE
  • MEMORY IMPAIRMENT
  • CONTROL SAMPLE
  • TEST SCORE
  • NEUROPSYCHOLOGY
  • DISSOCIATIONS
  • RECOGNITION
  • PERFORMANCE

Cite this

Methods of testing for a deficit in single-case studies: Evaluation of statistical power by Monte Carlo simulation. / Crawford, John Robertson; Garthwaite, P. H.

In: Cognitive Neuropsychology, Vol. 23, No. 6, 09.2006, p. 877-904.

Research output: Contribution to journalArticle

@article{a0c1f5df415c49b8bc4a2402c61b4d5a,
title = "Methods of testing for a deficit in single-case studies: Evaluation of statistical power by Monte Carlo simulation",
abstract = "Testing for the presence of a deficit by comparing a case to controls is a fundamental feature of many neuropsychological single-case studies. Monte Carlo simulation was employed to study the statistical power of two competing approaches to this task. The power to detect a large deficit was low to moderate for a method proposed by Crawford and Howell (1998; ranging from 44{\%} to 63{\%}) but was extremely low for a method proposed by Mycroft, Mitchell, and Kay (2002; ranging from 1{\%} to 13{\%}). The effects of departures from normality were examined, as was the effect of varying degrees of measurement error in the scores of controls and the single case. Measurement error produced a moderate reduction in power when present in both controls and the case; the effect of differentially greater measurement error for the single case depended on the initial level of power. When Mycroft et al.'s method was used to test for the presence of a classical dissociation, it produced very high Type I error rates (ranging from 20.7{\%} to 49.3{\%}); in contrast, the rates for criteria proposed by Crawford and Garthwaite (2005b) were low (ranging from 1.3{\%} to 6.7{\%}). The broader implications of these results for single-case research are discussed.",
keywords = "SKEW-NORMAL-DISTRIBUTION, T-DISTRIBUTION, ALZHEIMERS-DISEASE, MEMORY IMPAIRMENT, CONTROL SAMPLE, TEST SCORE, NEUROPSYCHOLOGY, DISSOCIATIONS, RECOGNITION, PERFORMANCE",
author = "Crawford, {John Robertson} and Garthwaite, {P. H.}",
year = "2006",
month = "9",
doi = "10.1080/02643290500538372",
language = "English",
volume = "23",
pages = "877--904",
journal = "Cognitive Neuropsychology",
issn = "0264-3294",
publisher = "Psychology Press Ltd",
number = "6",

}

TY - JOUR

T1 - Methods of testing for a deficit in single-case studies: Evaluation of statistical power by Monte Carlo simulation

AU - Crawford, John Robertson

AU - Garthwaite, P. H.

PY - 2006/9

Y1 - 2006/9

N2 - Testing for the presence of a deficit by comparing a case to controls is a fundamental feature of many neuropsychological single-case studies. Monte Carlo simulation was employed to study the statistical power of two competing approaches to this task. The power to detect a large deficit was low to moderate for a method proposed by Crawford and Howell (1998; ranging from 44% to 63%) but was extremely low for a method proposed by Mycroft, Mitchell, and Kay (2002; ranging from 1% to 13%). The effects of departures from normality were examined, as was the effect of varying degrees of measurement error in the scores of controls and the single case. Measurement error produced a moderate reduction in power when present in both controls and the case; the effect of differentially greater measurement error for the single case depended on the initial level of power. When Mycroft et al.'s method was used to test for the presence of a classical dissociation, it produced very high Type I error rates (ranging from 20.7% to 49.3%); in contrast, the rates for criteria proposed by Crawford and Garthwaite (2005b) were low (ranging from 1.3% to 6.7%). The broader implications of these results for single-case research are discussed.

AB - Testing for the presence of a deficit by comparing a case to controls is a fundamental feature of many neuropsychological single-case studies. Monte Carlo simulation was employed to study the statistical power of two competing approaches to this task. The power to detect a large deficit was low to moderate for a method proposed by Crawford and Howell (1998; ranging from 44% to 63%) but was extremely low for a method proposed by Mycroft, Mitchell, and Kay (2002; ranging from 1% to 13%). The effects of departures from normality were examined, as was the effect of varying degrees of measurement error in the scores of controls and the single case. Measurement error produced a moderate reduction in power when present in both controls and the case; the effect of differentially greater measurement error for the single case depended on the initial level of power. When Mycroft et al.'s method was used to test for the presence of a classical dissociation, it produced very high Type I error rates (ranging from 20.7% to 49.3%); in contrast, the rates for criteria proposed by Crawford and Garthwaite (2005b) were low (ranging from 1.3% to 6.7%). The broader implications of these results for single-case research are discussed.

KW - SKEW-NORMAL-DISTRIBUTION

KW - T-DISTRIBUTION

KW - ALZHEIMERS-DISEASE

KW - MEMORY IMPAIRMENT

KW - CONTROL SAMPLE

KW - TEST SCORE

KW - NEUROPSYCHOLOGY

KW - DISSOCIATIONS

KW - RECOGNITION

KW - PERFORMANCE

U2 - 10.1080/02643290500538372

DO - 10.1080/02643290500538372

M3 - Article

VL - 23

SP - 877

EP - 904

JO - Cognitive Neuropsychology

JF - Cognitive Neuropsychology

SN - 0264-3294

IS - 6

ER -