Offline Sentence Processing Measures for testing Readability with Users

Advaith Siddharthan, Napoleon Katsos

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

While there has been much work on computational models to predict readability based on the lexical, syntactic and discourse properties of a text, there are also interesting open questions about how computer generated text should be evaluated with target populations. In this paper, we compare two offline methods for evaluating sentence quality, magnitude estimation of acceptability judgements and sentence recall. These methods differ in the extent to which they can differentiate between surface level fluency and deeper comprehension issues. We find, most importantly, that the two correlate. Magnitude estimation can be run on the web without supervision, and the results can be analysed automatically. The sentence recall methodology is more resource intensive, but allows us to tease apart the fluency and comprehension issues that arise.
Original languageEnglish
Title of host publicationProceedings of the NAACL 2012 Workshop on Predicting and Improving Text Readability (PITR 2012)
PublisherACL
Pages17-24
Number of pages8
Publication statusPublished - 2012

Fingerprint

Testing
Syntactics
Processing

Cite this

Siddharthan, A., & Katsos, N. (2012). Offline Sentence Processing Measures for testing Readability with Users. In Proceedings of the NAACL 2012 Workshop on Predicting and Improving Text Readability (PITR 2012) (pp. 17-24). ACL.

Offline Sentence Processing Measures for testing Readability with Users. / Siddharthan, Advaith; Katsos, Napoleon.

Proceedings of the NAACL 2012 Workshop on Predicting and Improving Text Readability (PITR 2012). ACL, 2012. p. 17-24.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Siddharthan, A & Katsos, N 2012, Offline Sentence Processing Measures for testing Readability with Users. in Proceedings of the NAACL 2012 Workshop on Predicting and Improving Text Readability (PITR 2012). ACL, pp. 17-24.
Siddharthan A, Katsos N. Offline Sentence Processing Measures for testing Readability with Users. In Proceedings of the NAACL 2012 Workshop on Predicting and Improving Text Readability (PITR 2012). ACL. 2012. p. 17-24
Siddharthan, Advaith ; Katsos, Napoleon. / Offline Sentence Processing Measures for testing Readability with Users. Proceedings of the NAACL 2012 Workshop on Predicting and Improving Text Readability (PITR 2012). ACL, 2012. pp. 17-24
@inproceedings{c273897ab7794f3189616e041ca4c942,
title = "Offline Sentence Processing Measures for testing Readability with Users",
abstract = "While there has been much work on computational models to predict readability based on the lexical, syntactic and discourse properties of a text, there are also interesting open questions about how computer generated text should be evaluated with target populations. In this paper, we compare two offline methods for evaluating sentence quality, magnitude estimation of acceptability judgements and sentence recall. These methods differ in the extent to which they can differentiate between surface level fluency and deeper comprehension issues. We find, most importantly, that the two correlate. Magnitude estimation can be run on the web without supervision, and the results can be analysed automatically. The sentence recall methodology is more resource intensive, but allows us to tease apart the fluency and comprehension issues that arise.",
author = "Advaith Siddharthan and Napoleon Katsos",
year = "2012",
language = "English",
pages = "17--24",
booktitle = "Proceedings of the NAACL 2012 Workshop on Predicting and Improving Text Readability (PITR 2012)",
publisher = "ACL",

}

TY - GEN

T1 - Offline Sentence Processing Measures for testing Readability with Users

AU - Siddharthan, Advaith

AU - Katsos, Napoleon

PY - 2012

Y1 - 2012

N2 - While there has been much work on computational models to predict readability based on the lexical, syntactic and discourse properties of a text, there are also interesting open questions about how computer generated text should be evaluated with target populations. In this paper, we compare two offline methods for evaluating sentence quality, magnitude estimation of acceptability judgements and sentence recall. These methods differ in the extent to which they can differentiate between surface level fluency and deeper comprehension issues. We find, most importantly, that the two correlate. Magnitude estimation can be run on the web without supervision, and the results can be analysed automatically. The sentence recall methodology is more resource intensive, but allows us to tease apart the fluency and comprehension issues that arise.

AB - While there has been much work on computational models to predict readability based on the lexical, syntactic and discourse properties of a text, there are also interesting open questions about how computer generated text should be evaluated with target populations. In this paper, we compare two offline methods for evaluating sentence quality, magnitude estimation of acceptability judgements and sentence recall. These methods differ in the extent to which they can differentiate between surface level fluency and deeper comprehension issues. We find, most importantly, that the two correlate. Magnitude estimation can be run on the web without supervision, and the results can be analysed automatically. The sentence recall methodology is more resource intensive, but allows us to tease apart the fluency and comprehension issues that arise.

M3 - Conference contribution

SP - 17

EP - 24

BT - Proceedings of the NAACL 2012 Workshop on Predicting and Improving Text Readability (PITR 2012)

PB - ACL

ER -