The 2022 ReproGen Shared Task on Reproducibility of Evaluations in NLG: Overview and Results

Anya Belz; Anastasia Shimorina; Maja Popović; Ehud Reiter

The 2022 ReproGen Shared Task on Reproducibility of Evaluations in NLG: Overview and Results

Anya Belz, Anastasia Shimorina, Maja Popović, Ehud Reiter

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

Abstract

Against a background of growing interest in reproducibility in NLP and ML, and as part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP, we organised the second shared task on reproducibility of evaluations in NLG, ReproGen 2022. This paper describes the shared task, summarises results from the reproduction studies submitted, and provides further comparative analysis of the results. Out of six initial team registrations, we received submissions from five teams. Meta-analysis of the five reproduction studies revealed varying degrees of reproducibility, and allowed further tentative conclusions about what types of eval- uation tend to have better reproducibility.

Original language	English
Title of host publication	Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges
Place of Publication	Waterville, Maine, USA and virtual meeting
Publisher	Association for Computational Linguistics
Pages	43-51
Number of pages	9
Publication status	Published - 1 Jul 2022

Access to Document

Belz_etal_ACL_The_2022_ReproGen_VoR
© 1963–2023 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License. https://creativecommons.org/licenses/by/4.0/
Final published version, 219 KBLicence: CC BY

https://aclanthology.org/2022.inlg-genchal.8Licence: CC BY

Cite this

The 2022 ReproGen Shared Task on Reproducibility of Evaluations in NLG : Overview and Results. / Belz, Anya; Shimorina, Anastasia; Popović, Maja et al.

Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges. Waterville, Maine, USA and virtual meeting : Association for Computational Linguistics, 2022. p. 43-51.

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

Belz, A, Shimorina, A, Popović, M & Reiter, E 2022, The 2022 ReproGen Shared Task on Reproducibility of Evaluations in NLG: Overview and Results. in Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges. Association for Computational Linguistics, Waterville, Maine, USA and virtual meeting, pp. 43-51. <https://aclanthology.org/2022.inlg-genchal.8>

@inproceedings{1fad0b88cf2142ba9f4e6c25ccfeba54,

title = "The 2022 ReproGen Shared Task on Reproducibility of Evaluations in NLG: Overview and Results",

abstract = "Against a background of growing interest in reproducibility in NLP and ML, and as part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP, we organised the second shared task on reproducibility of evaluations in NLG, ReproGen 2022. This paper describes the shared task, summarises results from the reproduction studies submitted, and provides further comparative analysis of the results. Out of six initial team registrations, we received submissions from five teams. Meta-analysis of the five reproduction studies revealed varying degrees of reproducibility, and allowed further tentative conclusions about what types of eval- uation tend to have better reproducibility.",

author = "Anya Belz and Anastasia Shimorina and Maja Popovi{\'c} and Ehud Reiter",

year = "2022",

month = jul,

day = "1",

language = "English",

pages = "43--51",

booktitle = "Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges",

publisher = "Association for Computational Linguistics",

}

TY - GEN

T1 - The 2022 ReproGen Shared Task on Reproducibility of Evaluations in NLG

T2 - Overview and Results

AU - Belz, Anya

AU - Shimorina, Anastasia

AU - Popović, Maja

AU - Reiter, Ehud

PY - 2022/7/1

Y1 - 2022/7/1

N2 - Against a background of growing interest in reproducibility in NLP and ML, and as part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP, we organised the second shared task on reproducibility of evaluations in NLG, ReproGen 2022. This paper describes the shared task, summarises results from the reproduction studies submitted, and provides further comparative analysis of the results. Out of six initial team registrations, we received submissions from five teams. Meta-analysis of the five reproduction studies revealed varying degrees of reproducibility, and allowed further tentative conclusions about what types of eval- uation tend to have better reproducibility.

AB - Against a background of growing interest in reproducibility in NLP and ML, and as part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP, we organised the second shared task on reproducibility of evaluations in NLG, ReproGen 2022. This paper describes the shared task, summarises results from the reproduction studies submitted, and provides further comparative analysis of the results. Out of six initial team registrations, we received submissions from five teams. Meta-analysis of the five reproduction studies revealed varying degrees of reproducibility, and allowed further tentative conclusions about what types of eval- uation tend to have better reproducibility.

M3 - Published conference contribution

SP - 43

EP - 51

BT - Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges

PB - Association for Computational Linguistics

CY - Waterville, Maine, USA and virtual meeting

ER -

The 2022 ReproGen Shared Task on Reproducibility of Evaluations in NLG: Overview and Results

Abstract

Access to Document

Fingerprint

Cite this