A system to detect inconsistencies between a domain expert's different perspectives on (classification) tasks

Derek Sleeman, Andy Aiken, Laura Moss, John Kinsella, Malcolm Sim

Research output: Chapter in Book/Report/Conference proceedingChapter

3 Citations (Scopus)

Abstract

This paper discusses the range of knowledge acquisition, including machine learning, approaches used to develop knowledge bases for Intelligent Systems. Specifically, this paper focuses on developing techniques which enable an expert to detect inconsistencies in 2 (or more) perspectives that the expert might have on the same (classification) task. Further, the INSIGHT system has been developed to provide a tool which supports domain experts exploring, and removing, the inconsistencies in their conceptualization of a task. We report here a study of Intensive Care physicians reconciling 2 perspectives on their patients. The high level task which the physicians had set themselves was to classify, on a 5 point scale (A-E), the hourly reports produced by the Unit’s patient management system. The 2 perspectives provided to INSIGHT were an annotated set of patient records where the expert had selected the appropriate category to describe that snapshot of the patient, and a set of rules which are able to classify the various time points on the same 5-point scale.

Inconsistencies between these 2 perspectives are displayed as a confusion matrix; moreover INSIGHT then allows the expert to revise both the annotated datasets (correcting data errors, and/or changing the assigned categories) and the actual rule-set. Each expert achieved a very high degree of consensus between his refined knowledge sources (i.e., annotated hourly patient records and the rule-set). Further, the consensus between the 2 experts was ~95%. The paper concludes by outlining some of the follow-up studies planned with both INSIGHT and this general approach.
Original languageEnglish
Title of host publicationAdvances in Machine Learning II
Subtitle of host publicationDedicated to the memory of Professor Ryszard S. Michalski
EditorsJacek Koronacki, Zbigniew W. Ras, Slawomir T. Wierzchon, Janusz Kacprzyk
Place of PublicationBerlin, Germany
PublisherSpringer-Verlag
Pages293-314
Number of pages22
Volume263
ISBN (Print)3642051782 , 978-3642051784
DOIs
Publication statusPublished - 24 Dec 2009

Publication series

NameStudies in Computational Intelligence
PublisherSpringer-Verlag
Volume263
ISSN (Print)1860-949X

Fingerprint

Knowledge acquisition
Intelligent systems
Learning systems

Keywords

  • classification task
  • expertize capture
  • knowledge-based systems
  • refinement
  • medical informatics

Cite this

Sleeman, D., Aiken, A., Moss, L., Kinsella, J., & Sim, M. (2009). A system to detect inconsistencies between a domain expert's different perspectives on (classification) tasks. In J. Koronacki, Z. W. Ras, S. T. Wierzchon, & J. Kacprzyk (Eds.), Advances in Machine Learning II: Dedicated to the memory of Professor Ryszard S. Michalski (Vol. 263, pp. 293-314). (Studies in Computational Intelligence; Vol. 263). Berlin, Germany: Springer-Verlag. https://doi.org/10.1007/978-3-642-05179-1_14

A system to detect inconsistencies between a domain expert's different perspectives on (classification) tasks. / Sleeman, Derek; Aiken, Andy; Moss, Laura; Kinsella, John; Sim, Malcolm.

Advances in Machine Learning II: Dedicated to the memory of Professor Ryszard S. Michalski. ed. / Jacek Koronacki; Zbigniew W. Ras; Slawomir T. Wierzchon; Janusz Kacprzyk . Vol. 263 Berlin, Germany : Springer-Verlag, 2009. p. 293-314 (Studies in Computational Intelligence; Vol. 263).

Research output: Chapter in Book/Report/Conference proceedingChapter

Sleeman, D, Aiken, A, Moss, L, Kinsella, J & Sim, M 2009, A system to detect inconsistencies between a domain expert's different perspectives on (classification) tasks. in J Koronacki, ZW Ras, ST Wierzchon & J Kacprzyk (eds), Advances in Machine Learning II: Dedicated to the memory of Professor Ryszard S. Michalski. vol. 263, Studies in Computational Intelligence, vol. 263, Springer-Verlag, Berlin, Germany, pp. 293-314. https://doi.org/10.1007/978-3-642-05179-1_14
Sleeman D, Aiken A, Moss L, Kinsella J, Sim M. A system to detect inconsistencies between a domain expert's different perspectives on (classification) tasks. In Koronacki J, Ras ZW, Wierzchon ST, Kacprzyk J, editors, Advances in Machine Learning II: Dedicated to the memory of Professor Ryszard S. Michalski. Vol. 263. Berlin, Germany: Springer-Verlag. 2009. p. 293-314. (Studies in Computational Intelligence). https://doi.org/10.1007/978-3-642-05179-1_14
Sleeman, Derek ; Aiken, Andy ; Moss, Laura ; Kinsella, John ; Sim, Malcolm. / A system to detect inconsistencies between a domain expert's different perspectives on (classification) tasks. Advances in Machine Learning II: Dedicated to the memory of Professor Ryszard S. Michalski. editor / Jacek Koronacki ; Zbigniew W. Ras ; Slawomir T. Wierzchon ; Janusz Kacprzyk . Vol. 263 Berlin, Germany : Springer-Verlag, 2009. pp. 293-314 (Studies in Computational Intelligence).
@inbook{6bc82acf12c74dbdb3a9f58ae6941f41,
title = "A system to detect inconsistencies between a domain expert's different perspectives on (classification) tasks",
abstract = "This paper discusses the range of knowledge acquisition, including machine learning, approaches used to develop knowledge bases for Intelligent Systems. Specifically, this paper focuses on developing techniques which enable an expert to detect inconsistencies in 2 (or more) perspectives that the expert might have on the same (classification) task. Further, the INSIGHT system has been developed to provide a tool which supports domain experts exploring, and removing, the inconsistencies in their conceptualization of a task. We report here a study of Intensive Care physicians reconciling 2 perspectives on their patients. The high level task which the physicians had set themselves was to classify, on a 5 point scale (A-E), the hourly reports produced by the Unit’s patient management system. The 2 perspectives provided to INSIGHT were an annotated set of patient records where the expert had selected the appropriate category to describe that snapshot of the patient, and a set of rules which are able to classify the various time points on the same 5-point scale. Inconsistencies between these 2 perspectives are displayed as a confusion matrix; moreover INSIGHT then allows the expert to revise both the annotated datasets (correcting data errors, and/or changing the assigned categories) and the actual rule-set. Each expert achieved a very high degree of consensus between his refined knowledge sources (i.e., annotated hourly patient records and the rule-set). Further, the consensus between the 2 experts was ~95{\%}. The paper concludes by outlining some of the follow-up studies planned with both INSIGHT and this general approach.",
keywords = "classification task, expertize capture, knowledge-based systems, refinement, medical informatics",
author = "Derek Sleeman and Andy Aiken and Laura Moss and John Kinsella and Malcolm Sim",
year = "2009",
month = "12",
day = "24",
doi = "10.1007/978-3-642-05179-1_14",
language = "English",
isbn = "3642051782",
volume = "263",
series = "Studies in Computational Intelligence",
publisher = "Springer-Verlag",
pages = "293--314",
editor = "Jacek Koronacki and Ras, {Zbigniew W. } and Wierzchon, { Slawomir T. } and {Kacprzyk }, Janusz",
booktitle = "Advances in Machine Learning II",

}

TY - CHAP

T1 - A system to detect inconsistencies between a domain expert's different perspectives on (classification) tasks

AU - Sleeman, Derek

AU - Aiken, Andy

AU - Moss, Laura

AU - Kinsella, John

AU - Sim, Malcolm

PY - 2009/12/24

Y1 - 2009/12/24

N2 - This paper discusses the range of knowledge acquisition, including machine learning, approaches used to develop knowledge bases for Intelligent Systems. Specifically, this paper focuses on developing techniques which enable an expert to detect inconsistencies in 2 (or more) perspectives that the expert might have on the same (classification) task. Further, the INSIGHT system has been developed to provide a tool which supports domain experts exploring, and removing, the inconsistencies in their conceptualization of a task. We report here a study of Intensive Care physicians reconciling 2 perspectives on their patients. The high level task which the physicians had set themselves was to classify, on a 5 point scale (A-E), the hourly reports produced by the Unit’s patient management system. The 2 perspectives provided to INSIGHT were an annotated set of patient records where the expert had selected the appropriate category to describe that snapshot of the patient, and a set of rules which are able to classify the various time points on the same 5-point scale. Inconsistencies between these 2 perspectives are displayed as a confusion matrix; moreover INSIGHT then allows the expert to revise both the annotated datasets (correcting data errors, and/or changing the assigned categories) and the actual rule-set. Each expert achieved a very high degree of consensus between his refined knowledge sources (i.e., annotated hourly patient records and the rule-set). Further, the consensus between the 2 experts was ~95%. The paper concludes by outlining some of the follow-up studies planned with both INSIGHT and this general approach.

AB - This paper discusses the range of knowledge acquisition, including machine learning, approaches used to develop knowledge bases for Intelligent Systems. Specifically, this paper focuses on developing techniques which enable an expert to detect inconsistencies in 2 (or more) perspectives that the expert might have on the same (classification) task. Further, the INSIGHT system has been developed to provide a tool which supports domain experts exploring, and removing, the inconsistencies in their conceptualization of a task. We report here a study of Intensive Care physicians reconciling 2 perspectives on their patients. The high level task which the physicians had set themselves was to classify, on a 5 point scale (A-E), the hourly reports produced by the Unit’s patient management system. The 2 perspectives provided to INSIGHT were an annotated set of patient records where the expert had selected the appropriate category to describe that snapshot of the patient, and a set of rules which are able to classify the various time points on the same 5-point scale. Inconsistencies between these 2 perspectives are displayed as a confusion matrix; moreover INSIGHT then allows the expert to revise both the annotated datasets (correcting data errors, and/or changing the assigned categories) and the actual rule-set. Each expert achieved a very high degree of consensus between his refined knowledge sources (i.e., annotated hourly patient records and the rule-set). Further, the consensus between the 2 experts was ~95%. The paper concludes by outlining some of the follow-up studies planned with both INSIGHT and this general approach.

KW - classification task

KW - expertize capture

KW - knowledge-based systems

KW - refinement

KW - medical informatics

U2 - 10.1007/978-3-642-05179-1_14

DO - 10.1007/978-3-642-05179-1_14

M3 - Chapter

SN - 3642051782

SN - 978-3642051784

VL - 263

T3 - Studies in Computational Intelligence

SP - 293

EP - 314

BT - Advances in Machine Learning II

A2 - Koronacki, Jacek

A2 - Ras, Zbigniew W.

A2 - Wierzchon, Slawomir T.

A2 - Kacprzyk , Janusz

PB - Springer-Verlag

CY - Berlin, Germany

ER -