Surgeons' non-technical skills in the operating room

Reliability testing of the NOTSS behavior rating system

Steven Yule, Rhona Flin, Nicola Maran, David Rowley, George Youngson, Simon Paterson-Brown

Research output: Contribution to journalArticle

258 Citations (Scopus)

Abstract

Background: Previous research has shown that surgeons' intraoperative non-technical skills are related to surgical outcomes. The aim of this study was to evaluate the reliability of the NOTSS (Non-technical Skills for Surgeons) behavior rating system. Based on task analysis, the system incorporates five categories of skills for safe surgical practice (Situation Awareness, Decision Making, Task Management, Communication & Teamwork, and Leadership).

Methods: Consultant (attending) surgeons (n = 44) from five Scottish hospitals attended one of six experimental sessions and were trained to use the NOTSS system. They then used the system to rate consultant surgeons' behaviors in six simulated operating room scenarios that were presented using video. Surgeons' ratings of the behaviors demonstrated in each scenario were compared to expert ratings ("accuracy"), and assessed for inter-rater reliability and internal consistency.

Results: The NOTSS system had a consistent internal structure. Although raters had minimal training, rating "accuracy" for acceptable/unacceptable behavior was above 60% for all categories, with mean of 0.67 scale points difference from reference (expert) ratings (on 4-point scale). For inter-rater reliability, the mean values of within-group agreement (r (wg)) were acceptable for the categories Communication & Teamwork (.70), and Leadership (.72), but below a priori criteria for other categories. Intra-class correlation coefficients (ICC) indicated high agreement using average measures (values were .95-.99).

Conclusions: With the requisite training, the prototype NOTSS system could be used reliably by surgeons to observe and rate surgeons' behaviors. The instrument should now be tested for usability in the operating room.

Original languageEnglish
Pages (from-to)548-556
Number of pages9
JournalWorld Journal of Surgery
Volume32
Issue number4
Early online date8 Feb 2008
DOIs
Publication statusPublished - Apr 2008

Keywords

  • errors

Cite this

Surgeons' non-technical skills in the operating room : Reliability testing of the NOTSS behavior rating system. / Yule, Steven; Flin, Rhona; Maran, Nicola; Rowley, David; Youngson, George; Paterson-Brown, Simon.

In: World Journal of Surgery, Vol. 32, No. 4, 04.2008, p. 548-556.

Research output: Contribution to journalArticle

Yule, Steven ; Flin, Rhona ; Maran, Nicola ; Rowley, David ; Youngson, George ; Paterson-Brown, Simon. / Surgeons' non-technical skills in the operating room : Reliability testing of the NOTSS behavior rating system. In: World Journal of Surgery. 2008 ; Vol. 32, No. 4. pp. 548-556.
@article{432633a040dc4cc78c825de0bca99231,
title = "Surgeons' non-technical skills in the operating room: Reliability testing of the NOTSS behavior rating system",
abstract = "Background: Previous research has shown that surgeons' intraoperative non-technical skills are related to surgical outcomes. The aim of this study was to evaluate the reliability of the NOTSS (Non-technical Skills for Surgeons) behavior rating system. Based on task analysis, the system incorporates five categories of skills for safe surgical practice (Situation Awareness, Decision Making, Task Management, Communication & Teamwork, and Leadership). Methods: Consultant (attending) surgeons (n = 44) from five Scottish hospitals attended one of six experimental sessions and were trained to use the NOTSS system. They then used the system to rate consultant surgeons' behaviors in six simulated operating room scenarios that were presented using video. Surgeons' ratings of the behaviors demonstrated in each scenario were compared to expert ratings ({"}accuracy{"}), and assessed for inter-rater reliability and internal consistency. Results: The NOTSS system had a consistent internal structure. Although raters had minimal training, rating {"}accuracy{"} for acceptable/unacceptable behavior was above 60{\%} for all categories, with mean of 0.67 scale points difference from reference (expert) ratings (on 4-point scale). For inter-rater reliability, the mean values of within-group agreement (r (wg)) were acceptable for the categories Communication & Teamwork (.70), and Leadership (.72), but below a priori criteria for other categories. Intra-class correlation coefficients (ICC) indicated high agreement using average measures (values were .95-.99). Conclusions: With the requisite training, the prototype NOTSS system could be used reliably by surgeons to observe and rate surgeons' behaviors. The instrument should now be tested for usability in the operating room.",
keywords = "errors",
author = "Steven Yule and Rhona Flin and Nicola Maran and David Rowley and George Youngson and Simon Paterson-Brown",
year = "2008",
month = "4",
doi = "10.1007/s00268-007-9320-z",
language = "English",
volume = "32",
pages = "548--556",
journal = "World Journal of Surgery",
issn = "0364-2313",
publisher = "Springer New York",
number = "4",

}

TY - JOUR

T1 - Surgeons' non-technical skills in the operating room

T2 - Reliability testing of the NOTSS behavior rating system

AU - Yule, Steven

AU - Flin, Rhona

AU - Maran, Nicola

AU - Rowley, David

AU - Youngson, George

AU - Paterson-Brown, Simon

PY - 2008/4

Y1 - 2008/4

N2 - Background: Previous research has shown that surgeons' intraoperative non-technical skills are related to surgical outcomes. The aim of this study was to evaluate the reliability of the NOTSS (Non-technical Skills for Surgeons) behavior rating system. Based on task analysis, the system incorporates five categories of skills for safe surgical practice (Situation Awareness, Decision Making, Task Management, Communication & Teamwork, and Leadership). Methods: Consultant (attending) surgeons (n = 44) from five Scottish hospitals attended one of six experimental sessions and were trained to use the NOTSS system. They then used the system to rate consultant surgeons' behaviors in six simulated operating room scenarios that were presented using video. Surgeons' ratings of the behaviors demonstrated in each scenario were compared to expert ratings ("accuracy"), and assessed for inter-rater reliability and internal consistency. Results: The NOTSS system had a consistent internal structure. Although raters had minimal training, rating "accuracy" for acceptable/unacceptable behavior was above 60% for all categories, with mean of 0.67 scale points difference from reference (expert) ratings (on 4-point scale). For inter-rater reliability, the mean values of within-group agreement (r (wg)) were acceptable for the categories Communication & Teamwork (.70), and Leadership (.72), but below a priori criteria for other categories. Intra-class correlation coefficients (ICC) indicated high agreement using average measures (values were .95-.99). Conclusions: With the requisite training, the prototype NOTSS system could be used reliably by surgeons to observe and rate surgeons' behaviors. The instrument should now be tested for usability in the operating room.

AB - Background: Previous research has shown that surgeons' intraoperative non-technical skills are related to surgical outcomes. The aim of this study was to evaluate the reliability of the NOTSS (Non-technical Skills for Surgeons) behavior rating system. Based on task analysis, the system incorporates five categories of skills for safe surgical practice (Situation Awareness, Decision Making, Task Management, Communication & Teamwork, and Leadership). Methods: Consultant (attending) surgeons (n = 44) from five Scottish hospitals attended one of six experimental sessions and were trained to use the NOTSS system. They then used the system to rate consultant surgeons' behaviors in six simulated operating room scenarios that were presented using video. Surgeons' ratings of the behaviors demonstrated in each scenario were compared to expert ratings ("accuracy"), and assessed for inter-rater reliability and internal consistency. Results: The NOTSS system had a consistent internal structure. Although raters had minimal training, rating "accuracy" for acceptable/unacceptable behavior was above 60% for all categories, with mean of 0.67 scale points difference from reference (expert) ratings (on 4-point scale). For inter-rater reliability, the mean values of within-group agreement (r (wg)) were acceptable for the categories Communication & Teamwork (.70), and Leadership (.72), but below a priori criteria for other categories. Intra-class correlation coefficients (ICC) indicated high agreement using average measures (values were .95-.99). Conclusions: With the requisite training, the prototype NOTSS system could be used reliably by surgeons to observe and rate surgeons' behaviors. The instrument should now be tested for usability in the operating room.

KW - errors

U2 - 10.1007/s00268-007-9320-z

DO - 10.1007/s00268-007-9320-z

M3 - Article

VL - 32

SP - 548

EP - 556

JO - World Journal of Surgery

JF - World Journal of Surgery

SN - 0364-2313

IS - 4

ER -