Aggregating Crowdsourced Quantitative Claims

Additive and Multiplicative Models

Robin Wentao Ouyang*, Lance M. Kaplan, Alice Toniolo, Mani Srivastava, Timothy J. Norman

*Corresponding author for this work

Research output: Contribution to journalArticle

7 Citations (Scopus)

Abstract

Truth discovery is an important technique for enabling reliable crowdsourcing applications. It aims to automatically discover the truths from possibly conflicting crowdsourced claims. Most existing truth discovery approaches focus on categorical applications, such as image classification. They use the accuracy, i.e., rate of exactly correct claims, to capture the reliability of participants. As a consequence, they are not effective for truth discovery in quantitative applications, such as percentage annotation and object counting, where similarity rather than exact matching between crowdsourced claims and latent truths should be considered. In this paper, we propose two unsupervised Quantitative Truth Finders (QTFs) for truth discovery in quantitative crowdsourcing applications. One QTF explores an additive model and the other explores a multiplicative model to capture different relationships between crowdsourced claims and latent truths in different classes of quantitative tasks. These QTFs naturally incorporate the similarity between variables. Moreover, they use the bias and the confidence instead of the accuracy to capture participants' abilities in quantity estimation. These QTFs are thus capable of accurately discovering quantitative truths in particular domains. Through extensive experiments, we demonstrate that these QTFs outperform other state-of-the-art approaches for truth discovery in quantitative crowdsourcing applications and they are also quite efficient.

Original languageEnglish
Pages (from-to)1621-1634
Number of pages14
JournalIEEE Transactions on Knowledge and Data Engineering
Volume28
Issue number7
Early online date29 Feb 2016
DOIs
Publication statusPublished - 1 Jul 2016

Keywords

  • Crowdsourcing
  • truth discovery
  • quantitative task
  • probabilistic graphical model

Cite this

Aggregating Crowdsourced Quantitative Claims : Additive and Multiplicative Models. / Ouyang, Robin Wentao; Kaplan, Lance M.; Toniolo, Alice; Srivastava, Mani; Norman, Timothy J.

In: IEEE Transactions on Knowledge and Data Engineering, Vol. 28, No. 7, 01.07.2016, p. 1621-1634.

Research output: Contribution to journalArticle

Ouyang, Robin Wentao ; Kaplan, Lance M. ; Toniolo, Alice ; Srivastava, Mani ; Norman, Timothy J. / Aggregating Crowdsourced Quantitative Claims : Additive and Multiplicative Models. In: IEEE Transactions on Knowledge and Data Engineering. 2016 ; Vol. 28, No. 7. pp. 1621-1634.
@article{930ea81f2ba341a58bd9c2a56433f702,
title = "Aggregating Crowdsourced Quantitative Claims: Additive and Multiplicative Models",
abstract = "Truth discovery is an important technique for enabling reliable crowdsourcing applications. It aims to automatically discover the truths from possibly conflicting crowdsourced claims. Most existing truth discovery approaches focus on categorical applications, such as image classification. They use the accuracy, i.e., rate of exactly correct claims, to capture the reliability of participants. As a consequence, they are not effective for truth discovery in quantitative applications, such as percentage annotation and object counting, where similarity rather than exact matching between crowdsourced claims and latent truths should be considered. In this paper, we propose two unsupervised Quantitative Truth Finders (QTFs) for truth discovery in quantitative crowdsourcing applications. One QTF explores an additive model and the other explores a multiplicative model to capture different relationships between crowdsourced claims and latent truths in different classes of quantitative tasks. These QTFs naturally incorporate the similarity between variables. Moreover, they use the bias and the confidence instead of the accuracy to capture participants' abilities in quantity estimation. These QTFs are thus capable of accurately discovering quantitative truths in particular domains. Through extensive experiments, we demonstrate that these QTFs outperform other state-of-the-art approaches for truth discovery in quantitative crowdsourcing applications and they are also quite efficient.",
keywords = "Crowdsourcing, truth discovery, quantitative task, probabilistic graphical model",
author = "Ouyang, {Robin Wentao} and Kaplan, {Lance M.} and Alice Toniolo and Mani Srivastava and Norman, {Timothy J.}",
note = "ACKNOWLEDGMENTS This work was supported in part by the U.S. ARL and U.K. Ministry of Defense under Agreement Number W911NF-06-3-0001, and by the U.S. NSF under award CNS-1213140. Any opinions, findings and conclusions or recommenda- tions expressed in this material are those of the author(s) and do not necessarily reflect the views or represent the official policies of the NSF, the U.S. ARL, the U.S. Government,the U.K. Ministry of Defense or the U.K. Government. The U.S. and U.K. Governments are authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation hereon. This work was done when R. W. Ouyang was a postdoc at the University of California, Los Angeles, CA.",
year = "2016",
month = "7",
day = "1",
doi = "10.1109/TKDE.2016.2535383",
language = "English",
volume = "28",
pages = "1621--1634",
journal = "IEEE Transactions on Knowledge and Data Engineering",
issn = "1041-4347",
publisher = "IEEE COMPUTER SOC",
number = "7",

}

TY - JOUR

T1 - Aggregating Crowdsourced Quantitative Claims

T2 - Additive and Multiplicative Models

AU - Ouyang, Robin Wentao

AU - Kaplan, Lance M.

AU - Toniolo, Alice

AU - Srivastava, Mani

AU - Norman, Timothy J.

N1 - ACKNOWLEDGMENTS This work was supported in part by the U.S. ARL and U.K. Ministry of Defense under Agreement Number W911NF-06-3-0001, and by the U.S. NSF under award CNS-1213140. Any opinions, findings and conclusions or recommenda- tions expressed in this material are those of the author(s) and do not necessarily reflect the views or represent the official policies of the NSF, the U.S. ARL, the U.S. Government,the U.K. Ministry of Defense or the U.K. Government. The U.S. and U.K. Governments are authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation hereon. This work was done when R. W. Ouyang was a postdoc at the University of California, Los Angeles, CA.

PY - 2016/7/1

Y1 - 2016/7/1

N2 - Truth discovery is an important technique for enabling reliable crowdsourcing applications. It aims to automatically discover the truths from possibly conflicting crowdsourced claims. Most existing truth discovery approaches focus on categorical applications, such as image classification. They use the accuracy, i.e., rate of exactly correct claims, to capture the reliability of participants. As a consequence, they are not effective for truth discovery in quantitative applications, such as percentage annotation and object counting, where similarity rather than exact matching between crowdsourced claims and latent truths should be considered. In this paper, we propose two unsupervised Quantitative Truth Finders (QTFs) for truth discovery in quantitative crowdsourcing applications. One QTF explores an additive model and the other explores a multiplicative model to capture different relationships between crowdsourced claims and latent truths in different classes of quantitative tasks. These QTFs naturally incorporate the similarity between variables. Moreover, they use the bias and the confidence instead of the accuracy to capture participants' abilities in quantity estimation. These QTFs are thus capable of accurately discovering quantitative truths in particular domains. Through extensive experiments, we demonstrate that these QTFs outperform other state-of-the-art approaches for truth discovery in quantitative crowdsourcing applications and they are also quite efficient.

AB - Truth discovery is an important technique for enabling reliable crowdsourcing applications. It aims to automatically discover the truths from possibly conflicting crowdsourced claims. Most existing truth discovery approaches focus on categorical applications, such as image classification. They use the accuracy, i.e., rate of exactly correct claims, to capture the reliability of participants. As a consequence, they are not effective for truth discovery in quantitative applications, such as percentage annotation and object counting, where similarity rather than exact matching between crowdsourced claims and latent truths should be considered. In this paper, we propose two unsupervised Quantitative Truth Finders (QTFs) for truth discovery in quantitative crowdsourcing applications. One QTF explores an additive model and the other explores a multiplicative model to capture different relationships between crowdsourced claims and latent truths in different classes of quantitative tasks. These QTFs naturally incorporate the similarity between variables. Moreover, they use the bias and the confidence instead of the accuracy to capture participants' abilities in quantity estimation. These QTFs are thus capable of accurately discovering quantitative truths in particular domains. Through extensive experiments, we demonstrate that these QTFs outperform other state-of-the-art approaches for truth discovery in quantitative crowdsourcing applications and they are also quite efficient.

KW - Crowdsourcing

KW - truth discovery

KW - quantitative task

KW - probabilistic graphical model

U2 - 10.1109/TKDE.2016.2535383

DO - 10.1109/TKDE.2016.2535383

M3 - Article

VL - 28

SP - 1621

EP - 1634

JO - IEEE Transactions on Knowledge and Data Engineering

JF - IEEE Transactions on Knowledge and Data Engineering

SN - 1041-4347

IS - 7

ER -