Comparing OECD PISA Reading in English to Other Languages: Identifying Potential Sources of Non-Invariance

Mustafa Asil*, Gavin T.L. Brown

*Corresponding author for this work

Research output: Contribution to journalArticle

11 Citations (Scopus)

Abstract

The use of the Programme for International Student Assessment (PISA) across nations, cultures, and languages has been criticized. The key criticisms point to the linguistic and cultural biases potentially underlying the design of reading comprehension tests, raising doubts about the legitimacy of comparisons across economies. Our research focused on the type and magnitude of invariance or non-invariance in the PISA Reading Comprehension test by language, culture, and economic development relative to performance of the Australian English-speaking reference group used to develop the tests. Multi-Group Confirmatory Factor Analysis based on means and covariance structure (MACS) modeling was used to establish a dMACS effect size index for each economy for the degree of non-invariance. Only three wealthy, English-speaking countries had scalar invariance with Australia. Moderate or large effects were observed in just 31% of the comparisons. PISA index of economic, social and cultural status had a moderate inverse correlation with dMACS suggesting that socioeconomic resourcing of education played a significant role in measurement invariance, while educational practice and language factors seemed to play a further small role in non-invariance. Alternative approaches to reporting PISA results consistent with non-invariance are suggested
Original languageEnglish
Pages (from-to)71-93
Number of pages23
JournalInternational Journal of Testing
Volume16
Issue number1
Early online date18 Sep 2015
DOIs
Publication statusPublished - 2016

    Fingerprint

Keywords

  • measurement invariance (MI)
  • Programme for International Student Assessment (PISA)
  • reading literacy

Cite this