Sherlock

a Semi-Automatic Framework for Quiz Generation Using a Hybrid Semantic Similarity Measure

Chenghua Lin, Dong Liu, Wei Pang, Zhe Wang

Research output: Contribution to journalArticle

9 Citations (Scopus)
7 Downloads (Pure)

Abstract

In this paper, we present a semi-automatic system (Sherlock) for quiz generation using linked data and textual descriptions of RDF resources. Sherlock is distinguished from existing quiz generation systems in its generic framework for domain-independent quiz generation as well as in the ability of controlling the difficulty level of the generated quizzes. Difficulty scaling is non-trivial, and it is fundamentally related to cognitive science. We approach the problem with a new angle by perceiving the level of knowledge difficulty as a similarity measure problem and propose a novel hybrid semantic similarity measure using linked data. Extensive experiments show that the proposed semantic similarity measure outperforms four strong baselines with more than 47 % gain in clustering accuracy. In addition, we discovered in the human quiz test that the model accuracy indeed shows a strong correlation with the pairwise quiz similarity.
Original languageEnglish
Pages (from-to)667-679
Number of pages13
JournalCognitive Computation
Volume7
Issue number6
Early online date4 Aug 2015
DOIs
Publication statusPublished - Dec 2015

Fingerprint

Semantics
Cognitive Science
Aptitude
Cluster Analysis
Experiments

Keywords

  • quiz generation
  • linked data
  • RDF
  • educational games
  • semantic similarity
  • text analytics

Cite this

Sherlock : a Semi-Automatic Framework for Quiz Generation Using a Hybrid Semantic Similarity Measure. / Lin, Chenghua; Liu, Dong ; Pang, Wei; Wang, Zhe.

In: Cognitive Computation, Vol. 7, No. 6, 12.2015, p. 667-679.

Research output: Contribution to journalArticle

@article{06c71a7aedec4d95846c5ae5e80a3283,
title = "Sherlock: a Semi-Automatic Framework for Quiz Generation Using a Hybrid Semantic Similarity Measure",
abstract = "In this paper, we present a semi-automatic system (Sherlock) for quiz generation using linked data and textual descriptions of RDF resources. Sherlock is distinguished from existing quiz generation systems in its generic framework for domain-independent quiz generation as well as in the ability of controlling the difficulty level of the generated quizzes. Difficulty scaling is non-trivial, and it is fundamentally related to cognitive science. We approach the problem with a new angle by perceiving the level of knowledge difficulty as a similarity measure problem and propose a novel hybrid semantic similarity measure using linked data. Extensive experiments show that the proposed semantic similarity measure outperforms four strong baselines with more than 47 {\%} gain in clustering accuracy. In addition, we discovered in the human quiz test that the model accuracy indeed shows a strong correlation with the pairwise quiz similarity.",
keywords = "quiz generation, linked data, RDF, educational games, semantic similarity, text analytics",
author = "Chenghua Lin and Dong Liu and Wei Pang and Zhe Wang",
note = "Acknowledgments This work is supported by the BBC Connected Studio programme (http://www.bbc.co.uk/partnersandsuppliers/con nectedstudio/), the award made by the RCUK Digital Economy theme to the dot.rural Digital Economy Hub; award reference EP/G066051/1, the award made by UK Economic & Social Research Council (ESRC); award reference ES/M001628/1, National Natural Science Foundation of China (NSFC) under Grant No. 61373051, and the China National Science and Technology Pillar Program (Grant No. 2013BAH07F05). The authors would like to thank Ryan Hussey for the work on the user interface design and Tom Cass and James Ruston for the help in developing the Sherlock application. We are also grateful to Herm Baskerville for creating the editorial quizzes and Nava Tintarev for many helpful discussions on the human evaluation.",
year = "2015",
month = "12",
doi = "10.1007/s12559-015-9347-7",
language = "English",
volume = "7",
pages = "667--679",
journal = "Cognitive Computation",
issn = "1866-9956",
publisher = "Springer New York",
number = "6",

}

TY - JOUR

T1 - Sherlock

T2 - a Semi-Automatic Framework for Quiz Generation Using a Hybrid Semantic Similarity Measure

AU - Lin, Chenghua

AU - Liu, Dong

AU - Pang, Wei

AU - Wang, Zhe

N1 - Acknowledgments This work is supported by the BBC Connected Studio programme (http://www.bbc.co.uk/partnersandsuppliers/con nectedstudio/), the award made by the RCUK Digital Economy theme to the dot.rural Digital Economy Hub; award reference EP/G066051/1, the award made by UK Economic & Social Research Council (ESRC); award reference ES/M001628/1, National Natural Science Foundation of China (NSFC) under Grant No. 61373051, and the China National Science and Technology Pillar Program (Grant No. 2013BAH07F05). The authors would like to thank Ryan Hussey for the work on the user interface design and Tom Cass and James Ruston for the help in developing the Sherlock application. We are also grateful to Herm Baskerville for creating the editorial quizzes and Nava Tintarev for many helpful discussions on the human evaluation.

PY - 2015/12

Y1 - 2015/12

N2 - In this paper, we present a semi-automatic system (Sherlock) for quiz generation using linked data and textual descriptions of RDF resources. Sherlock is distinguished from existing quiz generation systems in its generic framework for domain-independent quiz generation as well as in the ability of controlling the difficulty level of the generated quizzes. Difficulty scaling is non-trivial, and it is fundamentally related to cognitive science. We approach the problem with a new angle by perceiving the level of knowledge difficulty as a similarity measure problem and propose a novel hybrid semantic similarity measure using linked data. Extensive experiments show that the proposed semantic similarity measure outperforms four strong baselines with more than 47 % gain in clustering accuracy. In addition, we discovered in the human quiz test that the model accuracy indeed shows a strong correlation with the pairwise quiz similarity.

AB - In this paper, we present a semi-automatic system (Sherlock) for quiz generation using linked data and textual descriptions of RDF resources. Sherlock is distinguished from existing quiz generation systems in its generic framework for domain-independent quiz generation as well as in the ability of controlling the difficulty level of the generated quizzes. Difficulty scaling is non-trivial, and it is fundamentally related to cognitive science. We approach the problem with a new angle by perceiving the level of knowledge difficulty as a similarity measure problem and propose a novel hybrid semantic similarity measure using linked data. Extensive experiments show that the proposed semantic similarity measure outperforms four strong baselines with more than 47 % gain in clustering accuracy. In addition, we discovered in the human quiz test that the model accuracy indeed shows a strong correlation with the pairwise quiz similarity.

KW - quiz generation

KW - linked data

KW - RDF

KW - educational games

KW - semantic similarity

KW - text analytics

U2 - 10.1007/s12559-015-9347-7

DO - 10.1007/s12559-015-9347-7

M3 - Article

VL - 7

SP - 667

EP - 679

JO - Cognitive Computation

JF - Cognitive Computation

SN - 1866-9956

IS - 6

ER -