Evidence for Communicative Compensation in Debt Advice with Reduced Multimodality

Nicole Andelic, Aidan Feeney, Gary McKeown

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Research has found that professional advice with empathy displays and signs of listening lead to more successful outcomes. These skills are typically displayed through visual nonverbal signals, whereas reduced multimodal contexts have to use other strategies to compensate for the lack of visual nonverbal information. Debt advice is often a highly emotional scenario but to date there has been no research comparing fully multimodal with reduced multimodal debt advice. The aim of the current study was to compare explicit emotional content (as expressed verbally) and implicit emotional content (as expressed through paralinguistic cues) in face to face (FTF) and telephone debt advice recordings. Twenty-two debt advice recordings were coded as emotional or functional and processed through emotion recognition software. The analysis found that FTF recordings included more explicit emotion than telephone recordings did. However, linear mixed effects modelling found substantially higher levels of arousal and slightly lower levels of valence in telephone advice. Interaction analyses found that emotional speech in FTF advice was characterised by lower levels of arousal than during functional speech, whereas emotional speech in telephone advice had higher levels of arousal than in functional speech. We can conclude that there are differences in emotional content when comparing full and reduced multimodal debt advice. Furthermore, as telephone advice cannot avail of visual nonverbal signals, it seems to compensate by using nonverbal cues present in the voice.
Original languageEnglish
Title of host publicationICMI '19 2019 International Conference on Multimodal Interaction
EditorsWen Gao, Helen Mei Ling Meng, Matthew Turk, Susan R Fussell, Bjorn Schuller, Yale Song, Kai Yu
Place of PublicationNew York
PublisherAssociation for Computing Machinery (ACM)
Pages210-219
Number of pages10
ISBN (Electronic)9781450368605
ISBN (Print)978-1-4503-6860-5
DOIs
Publication statusPublished - 14 Oct 2019
Event21st ACM International Conference on Multimodal Interaction
- Suzhou, Jiangsu, China
Duration: 14 Oct 201918 Oct 2019

Conference

Conference21st ACM International Conference on Multimodal Interaction
Abbreviated titleICMI
CountryChina
CitySuzhou, Jiangsu
Period14/10/1918/10/19

Fingerprint

Telephone
Arousal
Cues
Emotions
Research
Software

Keywords

  • Advice-giving
  • Affective computing
  • Arousal
  • Valence

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Hardware and Architecture
  • Computer Science Applications

Cite this

Andelic, N., Feeney, A., & McKeown, G. (2019). Evidence for Communicative Compensation in Debt Advice with Reduced Multimodality. In W. Gao, H. M. Ling Meng, M. Turk, S. R. Fussell, B. Schuller, Y. Song, & K. Yu (Eds.), ICMI '19 2019 International Conference on Multimodal Interaction (pp. 210-219). New York: Association for Computing Machinery (ACM). https://doi.org/10.1145/3340555.3353757

Evidence for Communicative Compensation in Debt Advice with Reduced Multimodality. / Andelic, Nicole; Feeney, Aidan; McKeown, Gary.

ICMI '19 2019 International Conference on Multimodal Interaction. ed. / Wen Gao; Helen Mei Ling Meng; Matthew Turk; Susan R Fussell; Bjorn Schuller; Yale Song; Kai Yu. New York : Association for Computing Machinery (ACM), 2019. p. 210-219.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Andelic, N, Feeney, A & McKeown, G 2019, Evidence for Communicative Compensation in Debt Advice with Reduced Multimodality. in W Gao, HM Ling Meng, M Turk, SR Fussell, B Schuller, Y Song & K Yu (eds), ICMI '19 2019 International Conference on Multimodal Interaction. Association for Computing Machinery (ACM), New York, pp. 210-219, 21st ACM International Conference on Multimodal Interaction
, Suzhou, Jiangsu, China, 14/10/19. https://doi.org/10.1145/3340555.3353757
Andelic N, Feeney A, McKeown G. Evidence for Communicative Compensation in Debt Advice with Reduced Multimodality. In Gao W, Ling Meng HM, Turk M, Fussell SR, Schuller B, Song Y, Yu K, editors, ICMI '19 2019 International Conference on Multimodal Interaction. New York: Association for Computing Machinery (ACM). 2019. p. 210-219 https://doi.org/10.1145/3340555.3353757
Andelic, Nicole ; Feeney, Aidan ; McKeown, Gary. / Evidence for Communicative Compensation in Debt Advice with Reduced Multimodality. ICMI '19 2019 International Conference on Multimodal Interaction. editor / Wen Gao ; Helen Mei Ling Meng ; Matthew Turk ; Susan R Fussell ; Bjorn Schuller ; Yale Song ; Kai Yu. New York : Association for Computing Machinery (ACM), 2019. pp. 210-219
@inproceedings{a67b26e48e934de48f20b9be2009df1e,
title = "Evidence for Communicative Compensation in Debt Advice with Reduced Multimodality",
abstract = "Research has found that professional advice with empathy displays and signs of listening lead to more successful outcomes. These skills are typically displayed through visual nonverbal signals, whereas reduced multimodal contexts have to use other strategies to compensate for the lack of visual nonverbal information. Debt advice is often a highly emotional scenario but to date there has been no research comparing fully multimodal with reduced multimodal debt advice. The aim of the current study was to compare explicit emotional content (as expressed verbally) and implicit emotional content (as expressed through paralinguistic cues) in face to face (FTF) and telephone debt advice recordings. Twenty-two debt advice recordings were coded as emotional or functional and processed through emotion recognition software. The analysis found that FTF recordings included more explicit emotion than telephone recordings did. However, linear mixed effects modelling found substantially higher levels of arousal and slightly lower levels of valence in telephone advice. Interaction analyses found that emotional speech in FTF advice was characterised by lower levels of arousal than during functional speech, whereas emotional speech in telephone advice had higher levels of arousal than in functional speech. We can conclude that there are differences in emotional content when comparing full and reduced multimodal debt advice. Furthermore, as telephone advice cannot avail of visual nonverbal signals, it seems to compensate by using nonverbal cues present in the voice.",
keywords = "Advice-giving, Affective computing, Arousal, Valence",
author = "Nicole Andelic and Aidan Feeney and Gary McKeown",
year = "2019",
month = "10",
day = "14",
doi = "10.1145/3340555.3353757",
language = "English",
isbn = "978-1-4503-6860-5",
pages = "210--219",
editor = "Wen Gao and {Ling Meng}, {Helen Mei} and Turk, {Matthew } and Fussell, {Susan R} and Bjorn Schuller and Yale Song and Yu, {Kai }",
booktitle = "ICMI '19 2019 International Conference on Multimodal Interaction",
publisher = "Association for Computing Machinery (ACM)",
address = "United States",

}

TY - GEN

T1 - Evidence for Communicative Compensation in Debt Advice with Reduced Multimodality

AU - Andelic, Nicole

AU - Feeney, Aidan

AU - McKeown, Gary

PY - 2019/10/14

Y1 - 2019/10/14

N2 - Research has found that professional advice with empathy displays and signs of listening lead to more successful outcomes. These skills are typically displayed through visual nonverbal signals, whereas reduced multimodal contexts have to use other strategies to compensate for the lack of visual nonverbal information. Debt advice is often a highly emotional scenario but to date there has been no research comparing fully multimodal with reduced multimodal debt advice. The aim of the current study was to compare explicit emotional content (as expressed verbally) and implicit emotional content (as expressed through paralinguistic cues) in face to face (FTF) and telephone debt advice recordings. Twenty-two debt advice recordings were coded as emotional or functional and processed through emotion recognition software. The analysis found that FTF recordings included more explicit emotion than telephone recordings did. However, linear mixed effects modelling found substantially higher levels of arousal and slightly lower levels of valence in telephone advice. Interaction analyses found that emotional speech in FTF advice was characterised by lower levels of arousal than during functional speech, whereas emotional speech in telephone advice had higher levels of arousal than in functional speech. We can conclude that there are differences in emotional content when comparing full and reduced multimodal debt advice. Furthermore, as telephone advice cannot avail of visual nonverbal signals, it seems to compensate by using nonverbal cues present in the voice.

AB - Research has found that professional advice with empathy displays and signs of listening lead to more successful outcomes. These skills are typically displayed through visual nonverbal signals, whereas reduced multimodal contexts have to use other strategies to compensate for the lack of visual nonverbal information. Debt advice is often a highly emotional scenario but to date there has been no research comparing fully multimodal with reduced multimodal debt advice. The aim of the current study was to compare explicit emotional content (as expressed verbally) and implicit emotional content (as expressed through paralinguistic cues) in face to face (FTF) and telephone debt advice recordings. Twenty-two debt advice recordings were coded as emotional or functional and processed through emotion recognition software. The analysis found that FTF recordings included more explicit emotion than telephone recordings did. However, linear mixed effects modelling found substantially higher levels of arousal and slightly lower levels of valence in telephone advice. Interaction analyses found that emotional speech in FTF advice was characterised by lower levels of arousal than during functional speech, whereas emotional speech in telephone advice had higher levels of arousal than in functional speech. We can conclude that there are differences in emotional content when comparing full and reduced multimodal debt advice. Furthermore, as telephone advice cannot avail of visual nonverbal signals, it seems to compensate by using nonverbal cues present in the voice.

KW - Advice-giving

KW - Affective computing

KW - Arousal

KW - Valence

UR - http://www.scopus.com/inward/record.url?scp=85074920693&partnerID=8YFLogxK

U2 - 10.1145/3340555.3353757

DO - 10.1145/3340555.3353757

M3 - Conference contribution

SN - 978-1-4503-6860-5

SP - 210

EP - 219

BT - ICMI '19 2019 International Conference on Multimodal Interaction

A2 - Gao, Wen

A2 - Ling Meng, Helen Mei

A2 - Turk, Matthew

A2 - Fussell, Susan R

A2 - Schuller, Bjorn

A2 - Song, Yale

A2 - Yu, Kai

PB - Association for Computing Machinery (ACM)

CY - New York

ER -