Applying Rule Extraction & Rule Refinement techniques to (Blackbox) Classifiers

Julius Cepukenas, Chenghua Lin, Derek Sleeman

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Black-box classifiers are able to classify unseen instances, once they have been trained on an appropriate (domain) dataset. Such classifiers have the advantage of being generally very efficient but the disadvantage of not being able to explain their processes to a user. For these reasons, over the last decade or so, a number of rule extraction algorithms have been developed which are able to extract a rule-set from classifiers. The focus of this project has been to re-implement a state-of-the-art rule extraction system, OSRE [1], and then to show that when the extracted rules are refined by the Knowledge Refinement system, FIXIT, that the refinement process, in virtually all cases, improves the fidelity of the refined rule-set when compared with the rule-set extracted by OSRE. A statistically significant difference between these two approaches has been demonstrated. We investigated 4 classifiers (2 blackbox (Neural Networks & SVM), 1 Bayesian classifier & 1 (Decision-Tree-based) whitebox) and 4 domains, so a total of 16 Classifier-Dataset combinations were considered. In only 1 case (6.25%) was the result slightly worse; 5 cases (31.25%) were the same (these could not be improved), and the remaining 10 cases (62.5%) show significant improvements. In the future, we intend using similar approaches to improve the accuracy of the classification; this study focuses on fidelity.
Original languageEnglish
Title of host publicationK-CAP 2015 Proceedings of the 8th International Conference on Knowledge Capture
PublisherACM
ISBN (Electronic)978-1-4503-3849-3
DOIs
Publication statusPublished - 2015
EventK-CAP 2015 - The 8th International Conference on Knowledge Capture - USA, New York, United States
Duration: 7 Oct 201510 Oct 2015

Conference

ConferenceK-CAP 2015 - The 8th International Conference on Knowledge Capture
CountryUnited States
CityNew York
Period7/10/1510/10/15

Fingerprint

Classifiers
Decision trees
Neural networks

Cite this

Cepukenas, J., Lin, C., & Sleeman, D. (2015). Applying Rule Extraction & Rule Refinement techniques to (Blackbox) Classifiers. In K-CAP 2015 Proceedings of the 8th International Conference on Knowledge Capture [27] ACM. https://doi.org/10.1145/2815833.2816950

Applying Rule Extraction & Rule Refinement techniques to (Blackbox) Classifiers. / Cepukenas, Julius; Lin, Chenghua; Sleeman, Derek.

K-CAP 2015 Proceedings of the 8th International Conference on Knowledge Capture. ACM, 2015. 27.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Cepukenas, J, Lin, C & Sleeman, D 2015, Applying Rule Extraction & Rule Refinement techniques to (Blackbox) Classifiers. in K-CAP 2015 Proceedings of the 8th International Conference on Knowledge Capture., 27, ACM, K-CAP 2015 - The 8th International Conference on Knowledge Capture, New York, United States, 7/10/15. https://doi.org/10.1145/2815833.2816950
Cepukenas J, Lin C, Sleeman D. Applying Rule Extraction & Rule Refinement techniques to (Blackbox) Classifiers. In K-CAP 2015 Proceedings of the 8th International Conference on Knowledge Capture. ACM. 2015. 27 https://doi.org/10.1145/2815833.2816950
Cepukenas, Julius ; Lin, Chenghua ; Sleeman, Derek. / Applying Rule Extraction & Rule Refinement techniques to (Blackbox) Classifiers. K-CAP 2015 Proceedings of the 8th International Conference on Knowledge Capture. ACM, 2015.
@inproceedings{49d4751574f349a88c073158eafe134b,
title = "Applying Rule Extraction & Rule Refinement techniques to (Blackbox) Classifiers",
abstract = "Black-box classifiers are able to classify unseen instances, once they have been trained on an appropriate (domain) dataset. Such classifiers have the advantage of being generally very efficient but the disadvantage of not being able to explain their processes to a user. For these reasons, over the last decade or so, a number of rule extraction algorithms have been developed which are able to extract a rule-set from classifiers. The focus of this project has been to re-implement a state-of-the-art rule extraction system, OSRE [1], and then to show that when the extracted rules are refined by the Knowledge Refinement system, FIXIT, that the refinement process, in virtually all cases, improves the fidelity of the refined rule-set when compared with the rule-set extracted by OSRE. A statistically significant difference between these two approaches has been demonstrated. We investigated 4 classifiers (2 blackbox (Neural Networks & SVM), 1 Bayesian classifier & 1 (Decision-Tree-based) whitebox) and 4 domains, so a total of 16 Classifier-Dataset combinations were considered. In only 1 case (6.25{\%}) was the result slightly worse; 5 cases (31.25{\%}) were the same (these could not be improved), and the remaining 10 cases (62.5{\%}) show significant improvements. In the future, we intend using similar approaches to improve the accuracy of the classification; this study focuses on fidelity.",
author = "Julius Cepukenas and Chenghua Lin and Derek Sleeman",
note = "ACKNOWLEDGMENTS Support for one of the authors (JC) has been provided by the Development Fund of the University of Aberdeen. Dr Wamberto Vasconcelos has provided comments on this project.",
year = "2015",
doi = "10.1145/2815833.2816950",
language = "English",
booktitle = "K-CAP 2015 Proceedings of the 8th International Conference on Knowledge Capture",
publisher = "ACM",

}

TY - GEN

T1 - Applying Rule Extraction & Rule Refinement techniques to (Blackbox) Classifiers

AU - Cepukenas, Julius

AU - Lin, Chenghua

AU - Sleeman, Derek

N1 - ACKNOWLEDGMENTS Support for one of the authors (JC) has been provided by the Development Fund of the University of Aberdeen. Dr Wamberto Vasconcelos has provided comments on this project.

PY - 2015

Y1 - 2015

N2 - Black-box classifiers are able to classify unseen instances, once they have been trained on an appropriate (domain) dataset. Such classifiers have the advantage of being generally very efficient but the disadvantage of not being able to explain their processes to a user. For these reasons, over the last decade or so, a number of rule extraction algorithms have been developed which are able to extract a rule-set from classifiers. The focus of this project has been to re-implement a state-of-the-art rule extraction system, OSRE [1], and then to show that when the extracted rules are refined by the Knowledge Refinement system, FIXIT, that the refinement process, in virtually all cases, improves the fidelity of the refined rule-set when compared with the rule-set extracted by OSRE. A statistically significant difference between these two approaches has been demonstrated. We investigated 4 classifiers (2 blackbox (Neural Networks & SVM), 1 Bayesian classifier & 1 (Decision-Tree-based) whitebox) and 4 domains, so a total of 16 Classifier-Dataset combinations were considered. In only 1 case (6.25%) was the result slightly worse; 5 cases (31.25%) were the same (these could not be improved), and the remaining 10 cases (62.5%) show significant improvements. In the future, we intend using similar approaches to improve the accuracy of the classification; this study focuses on fidelity.

AB - Black-box classifiers are able to classify unseen instances, once they have been trained on an appropriate (domain) dataset. Such classifiers have the advantage of being generally very efficient but the disadvantage of not being able to explain their processes to a user. For these reasons, over the last decade or so, a number of rule extraction algorithms have been developed which are able to extract a rule-set from classifiers. The focus of this project has been to re-implement a state-of-the-art rule extraction system, OSRE [1], and then to show that when the extracted rules are refined by the Knowledge Refinement system, FIXIT, that the refinement process, in virtually all cases, improves the fidelity of the refined rule-set when compared with the rule-set extracted by OSRE. A statistically significant difference between these two approaches has been demonstrated. We investigated 4 classifiers (2 blackbox (Neural Networks & SVM), 1 Bayesian classifier & 1 (Decision-Tree-based) whitebox) and 4 domains, so a total of 16 Classifier-Dataset combinations were considered. In only 1 case (6.25%) was the result slightly worse; 5 cases (31.25%) were the same (these could not be improved), and the remaining 10 cases (62.5%) show significant improvements. In the future, we intend using similar approaches to improve the accuracy of the classification; this study focuses on fidelity.

U2 - 10.1145/2815833.2816950

DO - 10.1145/2815833.2816950

M3 - Conference contribution

BT - K-CAP 2015 Proceedings of the 8th International Conference on Knowledge Capture

PB - ACM

ER -