Abstract
Despite two high profile series of challenges devoted to question answering technologies there remains no formal study into the representativeness that question corpora bear to real end-user inputs. We examine the corpora used presently and historically in the TREC and QALD challenges in juxtaposition with two more from natural sources and identify a degree of disjointedness between the two. We analyse these differences in depth before discussing a candidate approach to question corpora generation and provide a juxtaposition on its own representativeness. We conclude that these artificial corpora have good overall coverage of grammatical structures but the distribution is skewed, meaning performance measures may be inaccurate.
| Original language | English |
|---|---|
| Title of host publication | Information Access Evaluation. Multilinguality, Multimodality, and Interaction |
| Subtitle of host publication | CLEF 2014. |
| Editors | Evangelos Kanoulas, Mihai Lupu, Paul Clough, Mark Sanderson, Mark Hall, Allan Hanbury, Elaine Toms |
| Publisher | Springer-Verlag |
| Pages | 1-6 |
| Number of pages | 6 |
| ISBN (Electronic) | 9783319113821 |
| ISBN (Print) | 9783319113814 |
| DOIs | |
| Publication status | Published - 2014 |
| Event | 5th International Conference of the CLEF Initiative, CLEF 2014 - Sheffield, United Kingdom Duration: 15 Sep 2014 → 18 Sep 2014 |
Publication series
| Name | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
|---|---|
| Volume | 8685 LNCS |
| ISSN (Print) | 03029743 |
| ISSN (Electronic) | 16113349 |
Conference
| Conference | 5th International Conference of the CLEF Initiative, CLEF 2014 |
|---|---|
| Country | United Kingdom |
| City | Sheffield |
| Period | 15/09/14 → 18/09/14 |
Fingerprint
ASJC Scopus subject areas
- Computer Science(all)
- Theoretical Computer Science
Cite this
Making test corpora for question answering more representative. / Walker, Andrew; Starkey, Andrew; Pan, Jeff Z.; Siddharthan, Advaith.
Information Access Evaluation. Multilinguality, Multimodality, and Interaction : CLEF 2014. . ed. / Evangelos Kanoulas; Mihai Lupu; Paul Clough; Mark Sanderson; Mark Hall; Allan Hanbury; Elaine Toms. Springer-Verlag, 2014. p. 1-6 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8685 LNCS).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
}
TY - GEN
T1 - Making test corpora for question answering more representative
AU - Walker, Andrew
AU - Starkey, Andrew
AU - Pan, Jeff Z.
AU - Siddharthan, Advaith
PY - 2014
Y1 - 2014
N2 - Despite two high profile series of challenges devoted to question answering technologies there remains no formal study into the representativeness that question corpora bear to real end-user inputs. We examine the corpora used presently and historically in the TREC and QALD challenges in juxtaposition with two more from natural sources and identify a degree of disjointedness between the two. We analyse these differences in depth before discussing a candidate approach to question corpora generation and provide a juxtaposition on its own representativeness. We conclude that these artificial corpora have good overall coverage of grammatical structures but the distribution is skewed, meaning performance measures may be inaccurate.
AB - Despite two high profile series of challenges devoted to question answering technologies there remains no formal study into the representativeness that question corpora bear to real end-user inputs. We examine the corpora used presently and historically in the TREC and QALD challenges in juxtaposition with two more from natural sources and identify a degree of disjointedness between the two. We analyse these differences in depth before discussing a candidate approach to question corpora generation and provide a juxtaposition on its own representativeness. We conclude that these artificial corpora have good overall coverage of grammatical structures but the distribution is skewed, meaning performance measures may be inaccurate.
UR - http://www.scopus.com/inward/record.url?scp=84906776564&partnerID=8YFLogxK
U2 - 10.1007/978-3-319-11382-1_1
DO - 10.1007/978-3-319-11382-1_1
M3 - Conference contribution
SN - 9783319113814
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 1
EP - 6
BT - Information Access Evaluation. Multilinguality, Multimodality, and Interaction
A2 - Kanoulas, Evangelos
A2 - Lupu, Mihai
A2 - Clough, Paul
A2 - Sanderson, Mark
A2 - Hall, Mark
A2 - Hanbury, Allan
A2 - Toms, Elaine
PB - Springer-Verlag
ER -