Evaluating algorithms for the generation of referring expressions: going beyond toy domains

Ielka van der Sluis; Albert Gatt; Kees van Deemter

Evaluating algorithms for the generation of referring expressions: going beyond toy domains

Ielka van der Sluis^* (Corresponding Author), Albert Gatt, Kees van Deemter

^*Corresponding author for this work

Computing Science

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

Abstract

We describe a corpus-based evaluation methodology, applied to a number of classic algorithms in the generation of referring expressions. Following up on earlier work involving very simple domains, this paper deals with the issues associated with domains that contain ‘real-life’ objects of some complexity. Results indicate that state of the art algorithms perform very differently when applied to a complex domain. Moreover, if a version of the Incremental Algorithm is used then it becomes of huge importance to select a good preference order. These results should contribute to a growing debate on the evaluation of nlg systems, arguing in favour of carefully constructed balanced and semantically transparent corpora.

Original language	English
Title of host publication	Proceedings of Recent Advances in Natural Language Processing International Conference 2007
Editors	Galia Angelova, Kalina Bontcheva, Ruslan Mitkov, Nicolas Nicolov, Nicolai Nikolov
Place of Publication	Borovets, Bulgaria
Publisher	RANLP
Pages	541-547
Number of pages	7
ISBN (Electronic)	978-954-91743-7-3
Publication status	Published - 2007
Event	Recent Advances in Language Processing International Conference 2007 - Borovets, Russian Federation Duration: 27 Sep 2007 → 29 Sep 2007 https://lml.bas.bg/ranlp2007/

Conference

Conference	Recent Advances in Language Processing International Conference 2007
Abbreviated title	RANLP 2007
Country/Territory	Russian Federation
City	Borovets
Period	27/09/07 → 29/09/07
Internet address	https://lml.bas.bg/ranlp2007/

Keywords

generation of referring expressions
corpus-based evaluation of algorithms
Natural Language Generation

Access to Document

http://lml.bas.bg/ranlp2007/DOCS/RANLP2007.pdfLicence: Unspecified

Cite this

Sluis, I. V. D., Gatt, A., & van Deemter, K. (2007). Evaluating algorithms for the generation of referring expressions: going beyond toy domains. In G. Angelova, K. Bontcheva, R. Mitkov, N. Nicolov, & N. Nikolov (Eds.), Proceedings of Recent Advances in Natural Language Processing International Conference 2007 (pp. 541-547). RANLP. http://lml.bas.bg/ranlp2007/DOCS/RANLP2007.pdf

Evaluating algorithms for the generation of referring expressions : going beyond toy domains. / Sluis, Ielka van der (Corresponding Author); Gatt, Albert; van Deemter, Kees.

Proceedings of Recent Advances in Natural Language Processing International Conference 2007. ed. / Galia Angelova; Kalina Bontcheva; Ruslan Mitkov; Nicolas Nicolov; Nicolai Nikolov. Borovets, Bulgaria : RANLP, 2007. p. 541-547.

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

Sluis, IVD, Gatt, A & van Deemter, K 2007, Evaluating algorithms for the generation of referring expressions: going beyond toy domains. in G Angelova, K Bontcheva, R Mitkov, N Nicolov & N Nikolov (eds), Proceedings of Recent Advances in Natural Language Processing International Conference 2007. RANLP, Borovets, Bulgaria, pp. 541-547, Recent Advances in Language Processing International Conference 2007, Borovets, Russian Federation, 27/09/07. <http://lml.bas.bg/ranlp2007/DOCS/RANLP2007.pdf>

Sluis, Ielka van der ; Gatt, Albert ; van Deemter, Kees. / Evaluating algorithms for the generation of referring expressions : going beyond toy domains. Proceedings of Recent Advances in Natural Language Processing International Conference 2007. editor / Galia Angelova ; Kalina Bontcheva ; Ruslan Mitkov ; Nicolas Nicolov ; Nicolai Nikolov. Borovets, Bulgaria : RANLP, 2007. pp. 541-547

@inproceedings{67193f7a8f26461f830e85452638ccd0,

title = "Evaluating algorithms for the generation of referring expressions: going beyond toy domains",

abstract = "We describe a corpus-based evaluation methodology, applied to a number of classic algorithms in the generation of referring expressions. Following up on earlier work involving very simple domains, this paper deals with the issues associated with domains that contain {\textquoteleft}real-life{\textquoteright} objects of some complexity. Results indicate that state of the art algorithms perform very differently when applied to a complex domain. Moreover, if a version of the Incremental Algorithm is used then it becomes of huge importance to select a good preference order. These results should contribute to a growing debate on the evaluation of nlg systems, arguing in favour of carefully constructed balanced and semantically transparent corpora.",

keywords = "generation of referring expressions, corpus-based evaluation of algorithms, Natural Language Generation",

author = "Sluis, {Ielka van der} and Albert Gatt and {van Deemter}, Kees",

year = "2007",

language = "English",

pages = "541--547",

editor = "Galia Angelova and Kalina Bontcheva and Ruslan Mitkov and Nicolas Nicolov and Nicolai Nikolov",

booktitle = "Proceedings of Recent Advances in Natural Language Processing International Conference 2007",

publisher = "RANLP",

note = " Recent Advances in Language Processing International Conference 2007, RANLP 2007 ; Conference date: 27-09-2007 Through 29-09-2007",

url = "https://lml.bas.bg/ranlp2007/",

}

TY - GEN

T1 - Evaluating algorithms for the generation of referring expressions

T2 - Recent Advances in Language Processing International Conference 2007

AU - Sluis, Ielka van der

AU - Gatt, Albert

AU - van Deemter, Kees

PY - 2007

Y1 - 2007

N2 - We describe a corpus-based evaluation methodology, applied to a number of classic algorithms in the generation of referring expressions. Following up on earlier work involving very simple domains, this paper deals with the issues associated with domains that contain ‘real-life’ objects of some complexity. Results indicate that state of the art algorithms perform very differently when applied to a complex domain. Moreover, if a version of the Incremental Algorithm is used then it becomes of huge importance to select a good preference order. These results should contribute to a growing debate on the evaluation of nlg systems, arguing in favour of carefully constructed balanced and semantically transparent corpora.

AB - We describe a corpus-based evaluation methodology, applied to a number of classic algorithms in the generation of referring expressions. Following up on earlier work involving very simple domains, this paper deals with the issues associated with domains that contain ‘real-life’ objects of some complexity. Results indicate that state of the art algorithms perform very differently when applied to a complex domain. Moreover, if a version of the Incremental Algorithm is used then it becomes of huge importance to select a good preference order. These results should contribute to a growing debate on the evaluation of nlg systems, arguing in favour of carefully constructed balanced and semantically transparent corpora.

KW - generation of referring expressions

KW - corpus-based evaluation of algorithms

KW - Natural Language Generation

UR - https://lml.bas.bg/ranlp2007/

UR - https://www.academia.edu/616680/Evaluating_algorithms_for_the_generation_of_referring_expressions_Going_beyond_toy_domains

M3 - Published conference contribution

SP - 541

EP - 547

BT - Proceedings of Recent Advances in Natural Language Processing International Conference 2007

A2 - Angelova, Galia

A2 - Bontcheva, Kalina

A2 - Mitkov, Ruslan

A2 - Nicolov, Nicolas

A2 - Nikolov, Nicolai

PB - RANLP

CY - Borovets, Bulgaria

Y2 - 27 September 2007 through 29 September 2007

ER -

Evaluating algorithms for the generation of referring expressions: going beyond toy domains

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this