Barriers and enabling factors for error analysis in NLG research

Emiel Van Miltenburg; Miruna Clinciu; Ondřej Dušek; Dimitra Gkatzia; Stephanie Inglis; Leo Leppänen; Saad Mahamood; Stephanie Schoch; Craig Thomson; Luou Wen

doi:10.3384/nejlt.2000-1533.2023.4529

Barriers and enabling factors for error analysis in NLG research

Emiel Van Miltenburg, Miruna Clinciu, Ondřej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Stephanie Schoch, Craig Thomson, Luou Wen

Research output: Contribution to journal › Article › peer-review

Abstract

Earlier research has shown that few studies in Natural Language Generation (NLG) evaluate their system outputs using an error analysis, despite known limitations of automatic evaluation metrics and human ratings. This position paper takes the stance that error analyses should be encouraged, and discusses several ways to do so. This paper is not just based on our shared experience as authors, but we also distributed a survey as a means of public consultation. We provide an overview of existing barriers to carry out error analyses, and proposes changes to improve error reporting in the NLG literature.

Original language	English
Number of pages	22
Journal	Northern European Journal of Language Technology
Volume	9
Issue number	1
Early online date	21 Feb 2023
DOIs	https://doi.org/10.3384/nejlt.2000-1533.2023.4529
Publication status	Published - 21 Feb 2023

Access to Document

10.3384/nejlt.2000-1533.2023.4529Licence: CC BY

Miltenburg_etal_NEJLT_Barriers_And_Enabling_VoR
Copyright (c) 2023 Emiel van Miltenburg, Miruna Clinciu, Ondřej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Stephanie Schoch, Craig Thomson, Luou Wen This work is licensed under a Creative Commons Attribution 4.0 International License. https://creativecommons.org/licenses/by/4.0/
Final published version, 347 KBLicence: CC BY

Cite this

@article{080e0db4173a41b7aed7faddce7985f5,

title = "Barriers and enabling factors for error analysis in NLG research",

abstract = "Earlier research has shown that few studies in Natural Language Generation (NLG) evaluate their system outputs using an error analysis, despite known limitations of automatic evaluation metrics and human ratings. This position paper takes the stance that error analyses should be encouraged, and discusses several ways to do so. This paper is not just based on our shared experience as authors, but we also distributed a survey as a means of public consultation. We provide an overview of existing barriers to carry out error analyses, and proposes changes to improve error reporting in the NLG literature.",

author = "{Van Miltenburg}, Emiel and Miruna Clinciu and Ond{\v r}ej Du{\v s}ek and Dimitra Gkatzia and Stephanie Inglis and Leo Lepp{\"a}nen and Saad Mahamood and Stephanie Schoch and Craig Thomson and Luou Wen",

year = "2023",

month = feb,

day = "21",

doi = "10.3384/nejlt.2000-1533.2023.4529",

language = "English",

volume = "9",

journal = "Northern European Journal of Language Technology",

issn = "2000-1533",

number = "1",

}

TY - JOUR

T1 - Barriers and enabling factors for error analysis in NLG research

AU - Van Miltenburg, Emiel

AU - Clinciu, Miruna

AU - Dušek, Ondřej

AU - Gkatzia, Dimitra

AU - Inglis, Stephanie

AU - Leppänen, Leo

AU - Mahamood, Saad

AU - Schoch, Stephanie

AU - Thomson, Craig

AU - Wen, Luou

PY - 2023/2/21

Y1 - 2023/2/21

N2 - Earlier research has shown that few studies in Natural Language Generation (NLG) evaluate their system outputs using an error analysis, despite known limitations of automatic evaluation metrics and human ratings. This position paper takes the stance that error analyses should be encouraged, and discusses several ways to do so. This paper is not just based on our shared experience as authors, but we also distributed a survey as a means of public consultation. We provide an overview of existing barriers to carry out error analyses, and proposes changes to improve error reporting in the NLG literature.

AB - Earlier research has shown that few studies in Natural Language Generation (NLG) evaluate their system outputs using an error analysis, despite known limitations of automatic evaluation metrics and human ratings. This position paper takes the stance that error analyses should be encouraged, and discusses several ways to do so. This paper is not just based on our shared experience as authors, but we also distributed a survey as a means of public consultation. We provide an overview of existing barriers to carry out error analyses, and proposes changes to improve error reporting in the NLG literature.

U2 - 10.3384/nejlt.2000-1533.2023.4529

DO - 10.3384/nejlt.2000-1533.2023.4529

M3 - Article

VL - 9

JO - Northern European Journal of Language Technology

JF - Northern European Journal of Language Technology

SN - 2000-1533

IS - 1

ER -

Barriers and enabling factors for error analysis in NLG research

Abstract

Access to Document

Fingerprint

Cite this