Assessing the communication skills of doctors in training: reliability and sources of error

Medical examining bodies now commonly assess candidates' communication skills. However, there are a number of within- case sources of error that can mean examinations have poor reliability and validity. The aims of this study were to determine the main within- case sources of error and to identify the best methods of maximising reliability in highly structured communication skills assessments. Subjects were fifth year medical students at the University of Aberdeen. Subjects were videotaped giving clinical information to standardised patients and relatives toward the end of their fifth year and toward the end of their pre-registration year. Sources of variation, reliability coefficients and the most effective methods of reducing measurement error were calculated using generalisability theory. Systematic differences among subjects' communication skill was the main source of variation in two of the three assessment cases. The implications of the results are discussed.