Evaluating the effectiveness of explanations for recommender systems

Methodological issues and empirical studies on the impact of personalization

Research output: Contribution to journalArticle

95 Citations (Scopus)
4 Downloads (Pure)

Abstract

When recommender systems present items, these can be accompanied
by explanatory information. Such explanations can serve seven aims: effectiveness, satisfaction, transparency, scrutability, trust, persuasiveness, and efficiency. These aims can be incompatible, so any evaluation needs to state which aim is being investigated and use appropriate metrics. This paper focuses particularly on effectiveness (helping users to make good decisions) and its trade-off with satisfaction. It provides an overview of existing work on evaluating effectiveness and the metrics used. It also highlights the limitations of the existing effectiveness metrics, in particular the effects of under- and overestimation and recommendation domain. In addition to this methodological contribution, the paper presents four empirical studies in two domains: movies and cameras. These studies investigate the impact of personalizing simple feature-based explanations on effectiveness and satisfaction. Both approximated and real effectiveness is investigated. Contrary to expectation, personalization was detrimental to effectiveness, though it may improve user satisfaction. The studies also highlighted the importance of considering opt-out rates and the underlying rating distributionwhen evaluating effectiveness.
Original languageEnglish
Pages (from-to)399-439
Number of pages41
JournalUser Modelling and User-Adapted Interaction
Volume22
Issue number4-5
Early online date16 Feb 2012
DOIs
Publication statusPublished - Oct 2012

Fingerprint

Recommender systems
personalization
Transparency
Cameras
movies
transparency
rating
efficiency
present
evaluation

Keywords

  • recommender systems
  • metrics
  • item descriptions
  • explanations
  • empirical studies

Cite this

@article{e9c6b0a23a37436d8cb8d74430aa47be,
title = "Evaluating the effectiveness of explanations for recommender systems: Methodological issues and empirical studies on the impact of personalization",
abstract = "When recommender systems present items, these can be accompanied by explanatory information. Such explanations can serve seven aims: effectiveness, satisfaction, transparency, scrutability, trust, persuasiveness, and efficiency. These aims can be incompatible, so any evaluation needs to state which aim is being investigated and use appropriate metrics. This paper focuses particularly on effectiveness (helping users to make good decisions) and its trade-off with satisfaction. It provides an overview of existing work on evaluating effectiveness and the metrics used. It also highlights the limitations of the existing effectiveness metrics, in particular the effects of under- and overestimation and recommendation domain. In addition to this methodological contribution, the paper presents four empirical studies in two domains: movies and cameras. These studies investigate the impact of personalizing simple feature-based explanations on effectiveness and satisfaction. Both approximated and real effectiveness is investigated. Contrary to expectation, personalization was detrimental to effectiveness, though it may improve user satisfaction. The studies also highlighted the importance of considering opt-out rates and the underlying rating distributionwhen evaluating effectiveness.",
keywords = "recommender systems, metrics, item descriptions, explanations, empirical studies",
author = "Nava Tintarev and Judith Masthoff",
year = "2012",
month = "10",
doi = "10.1007/s11257-011-9117-5",
language = "English",
volume = "22",
pages = "399--439",
journal = "User Modelling and User-Adapted Interaction",
issn = "0924-1868",
publisher = "Springer Netherlands",
number = "4-5",

}

TY - JOUR

T1 - Evaluating the effectiveness of explanations for recommender systems

T2 - Methodological issues and empirical studies on the impact of personalization

AU - Tintarev, Nava

AU - Masthoff, Judith

PY - 2012/10

Y1 - 2012/10

N2 - When recommender systems present items, these can be accompanied by explanatory information. Such explanations can serve seven aims: effectiveness, satisfaction, transparency, scrutability, trust, persuasiveness, and efficiency. These aims can be incompatible, so any evaluation needs to state which aim is being investigated and use appropriate metrics. This paper focuses particularly on effectiveness (helping users to make good decisions) and its trade-off with satisfaction. It provides an overview of existing work on evaluating effectiveness and the metrics used. It also highlights the limitations of the existing effectiveness metrics, in particular the effects of under- and overestimation and recommendation domain. In addition to this methodological contribution, the paper presents four empirical studies in two domains: movies and cameras. These studies investigate the impact of personalizing simple feature-based explanations on effectiveness and satisfaction. Both approximated and real effectiveness is investigated. Contrary to expectation, personalization was detrimental to effectiveness, though it may improve user satisfaction. The studies also highlighted the importance of considering opt-out rates and the underlying rating distributionwhen evaluating effectiveness.

AB - When recommender systems present items, these can be accompanied by explanatory information. Such explanations can serve seven aims: effectiveness, satisfaction, transparency, scrutability, trust, persuasiveness, and efficiency. These aims can be incompatible, so any evaluation needs to state which aim is being investigated and use appropriate metrics. This paper focuses particularly on effectiveness (helping users to make good decisions) and its trade-off with satisfaction. It provides an overview of existing work on evaluating effectiveness and the metrics used. It also highlights the limitations of the existing effectiveness metrics, in particular the effects of under- and overestimation and recommendation domain. In addition to this methodological contribution, the paper presents four empirical studies in two domains: movies and cameras. These studies investigate the impact of personalizing simple feature-based explanations on effectiveness and satisfaction. Both approximated and real effectiveness is investigated. Contrary to expectation, personalization was detrimental to effectiveness, though it may improve user satisfaction. The studies also highlighted the importance of considering opt-out rates and the underlying rating distributionwhen evaluating effectiveness.

KW - recommender systems

KW - metrics

KW - item descriptions

KW - explanations

KW - empirical studies

U2 - 10.1007/s11257-011-9117-5

DO - 10.1007/s11257-011-9117-5

M3 - Article

VL - 22

SP - 399

EP - 439

JO - User Modelling and User-Adapted Interaction

JF - User Modelling and User-Adapted Interaction

SN - 0924-1868

IS - 4-5

ER -